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INTRODUCTION 



Evaluating workplace education programs: The 
current state of the art 

Evaluation is an organizational buzz word of the nineties. 
Schools, hospitals, businesses, private and publicly-funded social 
programs, even local, state, and federal governments are now talking 
about "quality standards," "continuous improvement," "indicators of 
success," and "cost-benefits." They do this in response to the call for 
quality management in all areas of national life, and, in a tight 
economy, to the need to justify to taxpayers and shareholders how 
their money is spent. 

Workplace basic skills programs are developing a similar 
vocabulary. In a climate of anxiety about the capacity of American 
business to compete in the global marketplace, workplace basic skills 
programs are being rapidly organized and funded ~ and then 
pressed by state, federal, and internal funders to determine if they 
are achieving desired results. 

Historically, evaluation of these programs has been done in two 
ways: (1) internal, informal monitoring by instructors( perhaps with 
some input from program participants) and (2) by outside 
evaluators. Each scenario has its strengths and limitations. 

Internal, informal evaluation has the advantage of providing in 
a timely way information which practitioners and learners can use to 
make sure the program is on track. One disadvantage of such 
internal evaluation has been that it often does not produce 
information deemed useful or reliable by other stakeholders like 
funders. 

External evaluators potentially have the expertise and 
objectivity to conduct evaluations which provide useful, reliable 
information. The disadvantage, however, of putting responsibility 
for the evaluation function primarily in the hands of an outsider is 
that stakeholders who should be involved in monitoring the program 
are, at best, passive recipients of information developed by others 
rather than active "investigators." Stakeholders who potentially have 
much to gain from and contribute to the program instead have less 
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understanding of the program's accomplishments and needs, and less 
"ownership" for the program. 

In situations where evaluation is required by outside sources 
(like funders) and carried out by external evaluators, program staff 
sometimes view evaluation as irrelevant or an imposition. They do 
so because they feel the standards which outsiders use to evaluate 
programs have little relevance to how their programs actually 
operate, what they achieve, or how they can be improved. 

Even if there is some general agreement that programs should 
focus primarily on improving worker job performance - and often 
there is no such agreement -- the content of instruction and 
definitions of success vary dramatically from site to site. In a 
developing field like workplace education, evaluations which assume 
that the same standards will apply equally in different settings will 
be flawed. On one hand, such evaluations will not capture the 
richness and diversity of program operations; on the other, they will 
present an inaccurate picture of what is being achieved. 

In situations where outside funding sources require program 
staff to generate information to report back to the funders, staff 
sometimes view evaluation as burdensome. Good evaluation takes 
time and requires skills and resources which staff either do not have 
or cannot spare. In most workplace basic skills programs, time and 
money are in short supply. Most educators, though skilled in 
assessing individual learners, are not trained in program evaluation, 
do not consider it part of their jobs, and are reluctant to take on 
evaluation without outside help. 

Team evaluation as an alternative 

The evaluation method presented in this Guide addresses these 
problems in existing approaches to evaluation. It attempts to 
combine the best aspects of current evaluation practice while 
avoiding its limitations. This Guide offers a way to produce high 
quality and relevant evaluations that can. be carried out - with 
reasonable cost - as part of normal program operations. 

In this approach, evaluations are conducted by "evaluation 
teams" composed of representatives of the stakeholder groups in 
your program. Stakeholders are all the people who have an interest 
or "stake" in making the program work, including participating 
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workers, teachers, supervisors, union leaders, and managers. The 
evaluation team, with the guidance of a team leader or facilitator, 
decides the evaluation's purpose and the questions it wants the 
evaluation to answer, collects and analyzes the necessary 
information, reports the findings, and uses those findings to improve 
program services. The evaluation can be as broad or focused as 
interest and resources allow. 

This collaborative approach is designed to build stakeholder 
understanding of the program and ownership and support for it. 
While strengthening the education program, this process also helps 
an organization make the transformation to the collaborative, 
continuous improvement approach to decision-making required by 
high-performance organizations. It does so by giving members an 
opportunity to develop a team identity and problem-solving skills 
which they can then apply to other tasks outside the education 
program. Over time, evaluation will become a natural activity within 
the education program, and collaborative decision-making will 
become a more natural part of day-to-day operations within the 
organization as a whole. 

Purposes of the Guide 

This Guide is designed to help the reader: 

• Understand basic evaluation concepts and procedures, 
especially as they relate to workplace basic skills programs. 

• Make decisions needed to plan and carry out all the steps 
of a team-based evaluation. 

• Cultivate an appreciation for evaluation and collaborative 
decision-making, so that they become integrated into the 
day-to-day operations and larger planning processes of both 
the education program and the organization as a whole. 

Audiences 

The primary audiences for this Guide are educators, union 
representatives, managers, supervisors, participating employees, 
funders, and others involved in workplace basic skills programs. The 
Guide might also be of interest to others outside the workplace basic 
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skills field who are looking for ways of getting stakeholders involved 
in planning and evaluating other kinds of activities carried out 
within workplaces, educational institutions, community organizations, 
and other contexts. 

How the Guide is structured 

The Guide is organized in five phases. Within each phase, the 
reader is presented with a series of steps and activities which an 
evaluation team may opt to undertake with greater or lesser 
intensity, depending on interests, resources, and experience. In this 
way, the Guide is more like a trail through a forest where you may 
step where you please, and linger or speed up at various points, as 
long as you don't lose sight of where you're going. In this way, the 
Guide is less like a recipe with exact proportions for all ingredients. 

The Guide demonstrates the decisions which should be made 
to carry out a meaningful evaluation. These decisions range from 
clarifying what you want to know to reporting your findings to 
targeted audiences. Readers who need more guidance on particular 
steps in the evaluation process -- such as how to design 
questionnaires or analyze statistical information - are encouraged to 
consult other resources (reference materials and others with special 
expertise) which can go into more depth in those particular areas. 

In Phase I . you -- the team facilitator -- will do some 
preparation work on your own and then organize your team. 
You and the team will consider your own prior experiences 
with evaluation and think through what you might achieve 
with a team evaluation. 

In Phase II. you and the team will identify audiences for the 
evaluation, decide what information you want to collect, and 
clarify how you might collect that information. 

In Phase III , you will design specific activities for collecting 
information and then field-test those activities. 

In Phase IV . the team will collect and analyze the information 
it wants, and then analyze and report its findings to the 
appropriate audiences. 



In Phase V, the team will figure out what actions need to be 
taken in response to feedback to the evaluation report. The 
team will also look back at the evaluation process itself, to 
decide whether and how it might continue evaluating the 
program in the future. 

Special features of this evaluation model 

Regardless of particular content or methodology, good 
evaluations meet several key criteria. They start with clear and 
relevant questions. They gather appropriate information efficiently. 
They interpret that information with insight. And they report their 
findings clearly in formats which are appropriate to their audiences. 

This Guide helps you to create an evaluation which meets these 
criteria. However, this particular team approach has several 
additional unique features. We believe that these features can 
increase your likelihood of producing a solid evaluation. They 
include: 

Active involvement of all stakeholders 

Formal evaluations are traditionally conducted by (1) an 
"outside" evaluator contracted to complete an evaluation for the 
program, and/or (2) an "internal" evaluator on the organization's 
staff who is given the job of monitoring the education program on 
an ongoing basis. 

In both cases, if they are good, the evaluators will work 
diligently to understand the program they are evaluating from the 
perspectives of the people who operate and participate in it. Too 
often, however, traditional evaluations do not adequately respond 
to the interests of stakeholders whose support is vital to make the 
program work. 

In our model, a team of people involved in the workplace 
education enterprise takes on the task of evaluating their own 
program, with guidance from a team leader or facilitator. This 
team is composed of representatives of the stakeholders who 
have invested time and/or other resources in the program, 
including worker-participants. 
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Conducting an evaluation with such a group can ensure that the 
needs and perspectives of the people closest to the program will 
be represented in all phases of the evaluation. In particular, 
requiring that the group include program participants makes it 
less likely that their particular needs and perspectives will be 
overlooked. 

A dual focus on "summative" and "formative" information 

Those involved in evaluations typically want to know what is 
being achieved in the program (that is, the "results" or 
"outcomes"). They want to know whether the program is meeting 
the goals which stakeholders have set and whether the program is 
also producing other, unanticipated results. This focus on "what's 
being achieved" is commonly referred to as "summative" (or 
"outcome") evaluation. 

Many stakeholders -- especially instructional staff but others 
responsible for program operations, as well -- also want to know 
whether various program components are working as they should. 
This focus on "what's working and what needs to be improved" 
goes by the name of "formative" evaluation. 

The team evaluation model presented here allows stakeholders 
to take an integrated approach to evaluation and focus on both 
summative and formative questions. 

In the "summative" side of its work, the team (1) clarifies 
program goals, (2) identifies "indicators" (or "evidence") which 
they would look for to determine whether those goals are being 
met, and (3) gathers and analyzes that evidence. In so doing, the 
team can determine whether the program is meeting 
stakeholders' expectations. The team can also capture any other 
outcomes which stakeholders hadn't originally anticipated. 

In its "formative" activities, the team (1) identifies the various 
program components that should be in place if program goals are 
to be met, (2) decides on "quality standards" for those components 
(that is, what each component would look like to work most 
effectively, (3) gathers and analyzes information which shows 
whether the components are meeting their quality standards, and 
(4) decides what actions are needed to improve the various 
program components. 
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By going through such an integrated process, stakeholders have 
a deeper understanding of what the program should be trying to 
accomplish, what is in fact being accomplished, and what is 
needed to make it work better. This integrated approach to 
evaluation is akin to the kinds of strategic planning models being 
proposed for the new, "high performance" workplace. 

The special leadership role of the team facilitator 

While this collaborative model encourages a sharing of 
responsibility for the evaluation process among a number of 
stakeholders, it does not mean that "no one is in charge." In fact, 
much of the work of conducting a team evaluation has to be done 
by one individual (or a small number of individuals). Such 
individuals ~ whom we call "facilitators" but others might refer to 
as "coordinators" or "team leaders" -- are vital and require certain 
leadership qualities, 

A facilitator must have (1) the time to devote to each step of 
the process, (2) the motivation (commitment or interest) to see 
that the evaluation process and the education program succeed, 
(3) a willingness to both share decision-making power and take 
responsibility for certain decisions and actions when needed, (4) 
some expertise in the "basics" of working with groups (for 
example, asking questions and listening carefully to what others 
say, organizing meetings, keeping track of information, etc.), and 
(5) the vision and courage to "stretch" oneself, try something new, 
and take some risks. 

The facilitator role might be played by: 

• One or more members of the evaluation team; 

• Someone from inside the host, organization who has some 

experience in conducting evaluations but is not 
necessarily associated with the workplace education 
program (like a union leader or human resource 
manager); 

• Someone from outside the organization, like an evaluation 
consultant. 

The "rank" the person brings to the role is not so important as a 
real interest in evaluation and a willingness and ability to 
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facilitate the necessary activities. Whoever initiates interest in 
the evaluation will ~ with other team members — have to decide 
if it is necessary to have a facilitator with formal experience in 
evaluation, or if the team members are ready to rely on their own 
resources and willingness to learn. For either situation, the Guide 
provides direction for the facilitator in how to provide support to 
the team. 

The Guide is written with the assumption that a team leader 
with little or no experience in evaluation is the primary reader of 
the Guide. However, all team members would benefit greatly 
from reading the Guide, to help them understand the process and 
the roles they can play. 

Modest vs. in-depth evaluation, "case studies" vs. 
"numbers": The team decides. 

When building a structure, a carpenter has a number of 
choices. Does she want "rough carpentry," as when building, say, a 
simple shed? Or is "fine cabinet-making" required, as when doing 
the finishing work on a house? 

Those involved in evaluating a workplace education program 
likewise have to make some decisions about the relative 
"precision" of the evaluation process to be used. While some 
might assume that the evaluation has to follow a precise, 
experimental model complete with carefully-refined procedures 
for gathering and analyzing information, in many situations such 
"precision" might simply not be necessary or practical. 

In this evaluation model, a sequence of evaluation activities is 
mapped out, but the scope and intensity of those activities are not 
prescribed. It is up to team members to decide what kinds of 
information they want, how much of it they need, and how it 
should be presented. These decisions will depend on (1) the 
audience for whom the evaluation is being prepared, (2) how 
much time team members can commit, and (3) how mucn prior 
experience team members have had with evaluation. 

Teams should realize that they can do everything from 
relatively simple, one-time "feedback sessions" with a sampling of 
stakeholders to a longer-term study of the program which relies 
on interviews, questionnaires, "tests," and other ways of collecting 



information. Information can be presented in a narrative, "case 
study" format which paints a portrait of the program using 
participants' own words and other rich evidence; or the team 
might want to go after "numbers" -- statistics which succinctly 
describe program impact and operations. 

Each of these ways of collecting and presenting information has 
strengths and limitations. Teams might decide to "get their feet 
wet" by trying out several different evaluation activities. They 
would view this initial foray not so much as a way of getting a 
precise measurement of the program but, rather, an opportunity 
to figure out where more-in-depth exploration might be useful 
and which ways of gathering and analyzing information are most 
feasible and meaningful. This Guide helps team members 
understand the options open to them and to decide which ones to 
use, given their particular situations. 

Evaluation as part of an ongoing process of strategic 
planning and continuous improvement 

Team-based evaluation assumes that evaluation is not a "one- 
shot deal." That is, as the program grows, the team will continue 
to generate relevant questions which are worthy of formal study. 
Therefore, the Guide encourages teams to start with questions 
which are manageable and whose answers will become building 
blocks for further evaluations. 

Evaluation is not seen as a one-time activity that answers all 
critical questions within a short time. While one round of 
evaluation activities may answer key questions of immediate 
importance, the Guide aims to make evaluation a tool of 
continuous improvement for the program. This means that 
evaluation is ongoing and that program staff use what they learn 
to improve program operations. 

Team evaluation also offers the opportunity to integrate 
evaluation activities and a team decision-making process with all 
other program planning functions (for example: organizational and 
individual needs analyses, curriculum design and implementation, 
learner recruitment and retention, staff development, and others). 
By going through a collaborative evaluation process, stakeholders 
will recognize the value of working together to plan and carry out 
various program components. Members will practice using team 
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decision-making skills which they can then apply elsewhere 
within both the education program and the larger organization. 

Supporting change in individuals, organizations, and 
education programs 

Many workplace educators are questioning some of the 
assumptions on which workplace education programs were built 
since the mid-1980s. 1 They note that new "high-performance" 
work organizations are supposed to emphasize shared decision- 
making, ongoing learning, and preparation for a range of jobs. 
This is a different kind of workplace than more traditional ones, in 
which workers were to follow orders, stay with the same job, and 
not spend too much time talking with co-workers. 

Given the changes in the workplace that we are presumably 
preparing workers for, many workplace educators question the 
relevance of some models of workplace education promoted since 
the mid-1980s. A growing number now feel that those earlier 
programs essentially prepared workers for the "old" kind of 
workplace, with decisions about program content made primarily 
by "education experts" and content focusing heavily on preparing 
workers to respond to written and oral instructions given by 
others. 

A new wave of workplace educators now argues for a different 
vision of workplace education which is more suitable for the new 
kind of workplace. In such a vision, decisions about education 
programs are made in the same collaborative way in which other 
workplace decisions are supposed to be made. Education is seen 
not only as a way of helping workers do their current jobs better, 
but to help them understand the larger workplace context, and 
figure out how they can improve that workplace and their own 
position in it. Education is thus seen not as a one-shot deal but as 
a tool for ongoing improvement of the organization and 
individuals. 

Those calling for this alternative vision have developed new 
ways of analyzing where basic education fits into the larger 



1 See Paul Jurmo's "Workplace Education: Stakeholders' Expectations, 
Practitioners' Responses, and the Role Evaluation Might Play," a review of 
selected literature on workplace education evaluation. 



organization. Education is not seen as something done for a few 
workers in isolation from the rest of the organization. Rather, 
througn a workplace needs assessment process, stakeholders first 
analyze the full range of obstacles preventing workers from 
participating fully in the improvement of the organization. A 
worker education program might be one of a number of initiatives 
the organization might take to eliminate those obstacles. 

An evaluation of a workplace education program would look 
not only at whether the education initiative is working; it would 
also examine whether the organization is successfully 
implementing the other initiatives to which it is committing itself. 
Worker job performance is thereby put in perspective, seen as 
dependent not only on what the learner and the education 
program do, but on whether other stakeholders are also doing 
their part to create conditions for high performance. 

Beyond standardized tests: Using diverse methods of 
collecting information 

When workplace education program stakeholders hear the 
word "evaluation," they often think "which test do we use?" This 
is an understandable reaction, given most people's experience in 
school where "evaluation" as equated with "tests." 

This Guide acknowledges that "tests" ~ both the standardized 
tests used in adult education programs and more-customized tests 
created around workplace-specific vocabulary and tasks ~ can be 
one way of measuring program impact. This Guide, however, 
challenges stakeholders to go beyond the knee-jerk response of 
"which test do we use" and instead think more deeply. To create a 
more-meaningful evaluation, stakeholders should first clarify 
what they expect the program to achieve, and then figure out how 
to determine whether those goals are in fact being met. A 
carefully-chosen or -customized "test" of some type (note that 
"tests" might include role-plays and other non-'paper-and-pencil" 
activities) might provide useful evidence for the team to consider. 
However, teams might also want to conduct interviews or focus 
groups with learners or supervisors or use other ways of 
documenting what participants are learning and how they are 
applying it. 




In addition to documenting changes in individual workers, an 
evaluation might want to document what the larger organization 
is doing to support learners as they try to apply what they are 
learning. As noted in the section directly above, if the larger 
organizational context does not help learners use what they are 
learning, the best efforts of the participants and instructors might 
have little impact on worker job performance. 

And, in addition to documenting changes in individuals and the 
organization, a comprehensive evaluation might also examine the 
various components which make up the program. Such 
"formative" evaluation would, for example, determine whether the 
procedures used for recruiting learners, designing instruction, or 
preparing instructors needed improvement. 

Evaluation as presented in this Guide is therefore much more 
than "assessing individual workers," whether through "tests" or 
other means. 

Practical advice 

Here are a few practical guidelines which can make your work 
easier and more enjoyable. 

1. Keep personal notes on all evaluation-related activities . 
Keep a notebook ~ an evaluation log — where you can jot 
down ideas and document the proceedings of team meetings 
and related activities. Keeping a log may come more easily 
to some people than others, but we strongly encourage you 
to develop the habit of documenting your evaluation 
activities. This will not only get you into the "evaluation 
mode" - a function which is largely observation, tiiinking, 
and documentation - but it will provide you with 
invaluable resource material as your evaluation grows. 

Documenting what happens in each team meeting is 
critical. Even brilliant ideas may fade within a few busy 
days. The team needs accurate reporting of its work so that 
nothing important is lost. Team members may take turns 
taking and distributing notes, but you should decide on a 
reporting format early on. Ensure that all team members 
receive minutes or some form of communication about what 
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happened at each team meetings. Start each meeting by 
reviewing what happened at the last meeting. 



* 2. Maintain confidentiality . Good evaluation requires that all 

involved have the freedom to say what is really on their 

I minds. If, in a team evaluation, confidentiality is not 

maintained, team members and those providing information 

. are liable to keep potentially-constructive ideas to 

I themselves, out of fear of being embarrassed or censured. 

An effective evaluation process will respect the privacy of 

| team discussions and of participants' responses. 

3. Respect stakeholders' time limitations . Meetings usually 
| take place on work time, so you will have to be very 

organized to get the work done in the time available. One 
^ option: Hold one two-hour meeting every other week for a 

I period of six months, with an additional two to three hours 

per week put in by the team coordinator or other team 
I members. Another option: Hold a series of half- or full-day 

I workshops in which the team moves through several 

evaluation tasks in one meeting. 

" Sometimes it will be impossible to carry out all activities 

l as a large group. To make best use of members' time, you 

[ might ask them to fill out questionnaires or do some other 

type of preparation before meetings. You might also break 
| the team into smaller groups to distribute tasks efficiently. 

4. Use visual aids . Use training tools (like flipcharts, 

I blackboards, overheads you can write on) in meetings to help 

■ keep everyone focused and "see" where the work is going. 

This may also serve as a record of the meeting to be used to 

I document the evaluation process afterward. 

I Develop a timeline with the team and keep it visible, 

| moving a marker from activity to activity as you complete 

your work. 

■ 5. Keep momentum g oin g . Don't let your team's initial 

enthusiasn wither. Each team meeting should be organized 
I to ensure that members feel they have accomplished 

something. To keep momentum going, make sure that 

■ everyone has an active role and can make a contribution. 
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Avoid making tasks too complicated or drawing out the 
team's activities over too many sessions. 

6. Enioy yourselves . Make the work engaging and enjoyable. 
For example, meet in a comfortable room at a time that is 
convenient for team members, have refreshments, or open 
sessions with a fun "ice-breaker." 

A final note 

This Guide is far from being the final word on how to evaluate 
workplace education programs. Evaluating these programs is 
complex and much work remains to be done to develop meaningful, 
efficient evaluation procedures. However, the Guide does provide a 
framework for those invested in workplace education to think about 
what workplace education programs might accomplish and how to 
understand what is in fact being achieved and needs to be done to 
strengthen programs. 

The authors will continue to work with others to refine this 
Guide and related materials. We hope to streamline the team 
evaluation process, record more-consistent examples of how a team 
might complete the many activities outlined in the Guide, and 
develop clearer procedures for gathering, analyzing, and reporting 
information. 

We welcome your feedback and questions. We would like to 
maintain a dialogue with you about how team evaluation works ~ or 
doesn't work ~ in your program. We may even create a computer 
network or other mechanisms so that those creating new approaches 
to workplace education practice what we're preaching in this Guide: 
ongoing communication, critical thinking, continuous improvement, 
learning, and collaboration. 

Please feel free to call or write. 
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PHASE I 

Prepare yourself 
and 

organize the team. 



What happens in Phase I 

In this fir.c phase, you will prepare yourself for your role as 
facilitator of a collaborative evaluation process, and you will organize 
your team. 

In Step 1, a series of questions introduces you to the 
fundamentals of team-based evaluation. 

In Step 2, you will ensure that a team is in place, and ready, 
willing, and able to do the work of planning and carrying out a high- 
quality evaluation. 
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Step 1 : Do your homework 



As the evaluation facilitator, you play a key role in making the 
evaluation work. You will need to prepare for the many tasks that 
lie ahead. 



ACTIVITY #l.a. 

ON YOUR OWN : THINK ABOUT WHO MIGHT JOIN YOUR 
TEAM, AND ANTICIPATE THEIR RESPONSES TO THE 
QUESTIONS WHICH LIE AHEAD. 

Read through this Guide, and note ideas and questions to return 
to later. Then take the time to answer the questions listed below. 
You might answer these questions in your "evaluation log," in 
conversation with one or two other people who support your 
efforts to initiate an evaluation, or in more-formal interviews with 
key program stakeholders. Take notes which will help you to plan 
subsequent parts of the evaluation. 

1. Is there an already-established group responsible for 
planning and overseeing your program? This could be a 
planning or advisory committee or a few people scattered 
throughout the organization who support your program. If so, 
who are the key players and what have they done so far? 
Have they conducted a workplace needs assessment or 
collected any other information about what the program should 
achieve or is achieving? What have they learned from the 
resulting information and from the process of gathering it? 
Are members of this group likely to support or participate in a 
team-based evaluation? 

2. Who are the stakeholder groups in vour program ? Who are 
the "stakeholders" who have invested time, money, or other 
resources in your program? Who in these groups might be 
interested in participating in a team-based evaluation? 

3. What do these stakeholders - including you - expect the 
program to achieve ? Do those involved see the program as a 
tool for enhancing workers' job performance, an education 
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benefit for workers' personal development, some combination 
of the two, or something else? 

4. What Questions do you think these stakeholders might want 
an evaluation to answer ? What might they want to know 
about the program? 

5. What are these stakeholders likely to do with the 
information gathered in an evaluation ? Are there particular 
purposes that they might want the evaluation to serve ~ for 
example, facilitate curriculum development or clarify whether 
the program should be expanded? 

6. Given the "culture" your program and the context it operates 
in. what kinds of information and information-gathering 
activities are stakeholders most likely to consider valuable? 
Will stakeholders be most interested in "numbers" and 
"statistics"? Or will they prefer less-structured but more- 
descriptive feedback from those involved in the program? Are 
paper-and-pencil tests or questionnaires suitable in this 
situation? Or might interviews and focus groups be better 
ways of getting information from program participants and 
others with knowledge of the program? 

7. What roles might stakeholders play in gathering, analyzing . 
and reporting information ? For the various stakeholders: 

• How much time are they likely to give to an evaluation? 

• Besides their time, what other resources might they 
contribute to an evaluation (e.g., expertise in conducting 
interviews, contacts, knowledge of the workplace, funds, 
meeting facilities, computer equipment)? 

• How interested will they likely be in using evaluation 
results to improve the program? 

• How willing and able are they to maintain confidentiality 
and work in a team? 

You have just thought through some key questions you will be 
discussing with your team during the first few steps of the 
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evaluation. Now take some time to analyze your personal 
experiences with evaluation. 



ACTIVITY #l.b. 

ON YOUR OWN : REFLECT ON YOUR PERSONAL EXPERIENCES 
WITH EVALUATION. 

Your prior experiences with evaluation will influence your 
expectations for this evaluation in both subtle and obvious ways. 
Jot down responses to the following questions to help you 
understand your experiences both "as an evaluator" and "being 
evaluated." 

Your experience as an "evaluator" 

1. In day-to-day situations 

On an average day, most of us evaluate ~ make 
judgments and decisions about ~ many different things, 
from what we eat to how others behave toward us. Think 
about how and in what situations you informally "evaluate" 
your own and others' behavior or performance ~ at work, in 
your family, or with friends. 



What do you typically evaluate? Do you evaluate what 
you eat, the clothes you put on each morning, how 
someone expresses an idea, or how someone gives you 
an instruction? 

Whose standards are you "evaluating" against ? Have 
you developed your own standards for good behavior 
or good taste? Or have you assumed standards set by 
someone else? 

What do vou do with the judg ments vou form from 
your informal evaluations ? How do you use them to 
guide your future actions? For example, after listening 
to a boring presentation, you might decide never to 
make a presentation without using visual aids. 
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2. In professional situations 

What, if any, more-formal evaluation activities have you 
carried out before? For example, in this or another program 
or in another job: 

• Who was responsible for conducting those evaluation 
activities? (One individual or a group?) 

• What were the purposes of the evaluations? 

• What kinds of information was collected? 

• Who set the standards which guided the evaluation? 

• How was the information used and by whom? 

• What were the strengths of the experiences? 

• What problems did you encounter? 

• What would you do differently? Why? 
Your experience "being evaluated" 

1. In what situations have you "been evaluated"? (Consider the 
spectrum ~ from getting a report card to a job-related 
performance appraisal.) 

2. In those situations: 

• Who conducted the evaluation? 

• What information did they collect about you? 

• Did you help to set the standards you were evaluated 
by? If no: How might the experience have been 
different if you had helped to set the standards? 

• Who used the information collected about you? 
For what purpose? 
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• How did you feel "being evaluated"? 

• What were the strengths of that experience? What was 
accomplished? Did you gain anything? 

• What problems did you encounter? 

• What would you do differently? Why? 



ACTIVITY #l.c. 

ON YOUR OWN : CONSIDER OTHER WAYS OF PREPARING 
YOURSELF FOR THIS EVALUATION. 

Here are some other things you might do to prepare yourself 
for the work ahead: 

• Review the Guide's bibliography. Locate and read the items that 

are most relevant to you. 

• Confer with others in or outside your organization who have 

conducted evaluations. Ask them to analyze their experience 
by responding to the kinds of questions presented under 
Activity #l.b. 

• Read evaluation reports which other workplace education 

programs have produced, or which other types of programs 
(e.g., technical training programs, cultural sensitivity 
workshops, etc.) in your workplace have produced. 

• Talk directly with others who have participated in team 

evaluations in particular, to ask them about their own 
experience and for guidance on questions you might have. 



By going through Acuities #l.a., #l.b., and #l.c, you will have 
thought through many of the issues you will face during the early 
stages of your evaluation. By talking in a preliminary way with 
other stakeholders, you will introduce them to the idea of 
participating in an evaluation team and get them to begin thinking 
about the questions they will face, as well. 
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Step 2: Organize the evaluation team. 



ACTIVITY #2.a. 

WITH KEY CONTACTS : IDENTIFY POTENTIAL TEAM 
MEMBERS. 

The evaluation team should be composed of representatives of 
all stakeholder groups in your program, including participating 
workers. The team may be essentially the same as a planning or 
advisory committee which oversees the program, a subcommittee 
of that planning group, or an entirely new group convened 
expressly for the evaluation. 

If you have a collaborative planning structure already in place 
in your program, you will have an informed, committed, and 
representative group to work with when you begin your 
evaluation. However, in programs where no such group has 
existed before, you will either have to recruit people you already 
know and think will make good team members; or work with a 
few key management, union, and/or worker contacts to decide 
who might serve on the evaluation team. 

Regardless of how you choose your team members, they should 
generally meet the following guidelines. (Note that these 
standards are almost identical to the criteria for team facilitators 
described in the Introduction.) 

Guidelines for Selecting Team Members 

1. A mix of workplace roles : Members should come from all 
company departments being served by the education program. 
Stakeholder categories include program participants, higher 
and supervisory-level managers, union representatives, 
education staff, and perhaps funders. The team should also 
have a representative mix of men and women, races and 
cultures, and ages. There should be at least two worker 
participants to ensure adequate representation of their 
perspectives and to provide mutual support. 
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2. A commitment to improving the education program : Team 
members should support the workplace education program and 
be open to using evaluation as a tool for improving it. 

3. Time : Members need time to participate actively in the team 
process, although not every member needs to participate in 
every activity. 

4. A willingness to work in a team : Members need to be willing 
to share decision making with other interest groups, some of 
whom they are not accustomed to sharing power with. 

5. A willing ness to try something new: Members should be 
willing to take some risks and develop new expertise in 
designing data-gathering activities, using them, analyzing data, 
and reporting findings. 

Potential team members need to make informed decisions 
about whether to become active members or not. Discussion 
about the purposes of evaluation and practical information about 
tasks and timelines will enable them to do so. Meet individually 
with potential team members to discuss what you hope a 
collaborative evaluation might accomplish and what roles they 
might play. If you are working with an already-established 
group, all or part of a regularly-scheduled meeting can be set 
aside to address these issues. In both cases, you might cover the 
following: 



ACTIVITY #2.b. 

WITH "CANDIDATES" FOR THE TEAM : CLARIFY PURPOSES 
AND GROUNDRULES FOR THE EVALUATION. 

In your meetings with potential team members: 

Discuss the "why's" of the evaluation. 

• Why would the program benefit from conducting an 
evaluation at this time? Brainstorm all possible benefits. 
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• What advantages will conducting an evaluation as a team 

have in particular? 

• What questions or problems that the program is currently 

dealing with might an evaluation address? 

• What questions might other audiences ~ for example, 

potential funders, union headquarters, or other departments 
in the organization - have that an evaluation might 
address? 

• Are there any disadvantages to conducting an evaluation at 

this time? 

Discuss the "how's and what's" of the evaluation. 

• What are the steps an evaluation team would go through? 
(You might display a flipchart or handout outlining the steps 
described in this Guide. Or give copies of the Guide to 
potential team members. Or copy relevant sections for them 
to review.) 

• What is the time commitment that team members need to 
make to conduct an initial round of evaluation activities? 
(Present a possible timeline.) 

• What questions do potential team members have about their 
roles and responsibilities in the evaluation? 

• What problems, if any, do potential team members foresee in 
conducting an evaluation and how might those problems be 
addressed? 

• Having considered all the above, do potential team members 
want to become part of the team? 

• Are there other people who might be good candidates for . 
membership on the team? 

By discussing these questions and any others which they may 
have, potential members will get to know each other better 
(especially important if they haven't worked together before), 
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better understand what might be accomplished, and come to a 
decision about who will participate on the team. 



Ideally, you will "hit the jackpot" at this early stage and 
smoothly pull together a group ready, willing, and able to get going 
with the evaluation. Or you might have to do a fair amount of 
explaining, "massaging," and negotiating with various members to 
clarify expectations, generate interest, and identify who is willing 
and able to participate. In some cases, you might not be able to pull 
together the broad group you want. You might have to work with 
only a few key people. 

Be aware that, if your organization does not typically conduct 
its work in teams, you will be asking evaluation team members to 
take on collaborative decision-making roles they might have little or 
no experience with. Some people might feel confused or intimidated 
at the prospect of working closely with others, especially if those 
others are supervisors or managers who oversee their work on the 
floor. Get these concerns out in the open and — as much as possible 
— resolved at this early stage, with humor and tact. These issues 
may continue to be worked on throughout the evaluation, but early 
emphasis on the need for mutual respect and trust will ensure a 
better evaluation and a better education program. 

You might also conclude that you cannot find a group which can 
develop and carry out a team evaluation. In such a case, rather than 
push people to do something they aren't ready to do, you should 
postpone the team evaluation project, at least until a time when a 
group can be formed with the right elements in place. 



ACTIVITY #2.c. 

WITH YOUR NEW MEMBERS : BUILD YOUR TEAM. 

In programs where a planning group has been in place for 
some time, it might not be necessary to carry out special team- 
building activities. However, for those sites where no planning 
group has been in place before ~ or where some new members 
are being added to an existing team ~ your new team members 
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might not have formed a true team identity yet and probably still 
have questions about what they will be doing. 

Consider whether, in the particular culture of your program, it 
would help to take some extra time at this early stage to create a 
foundation of common will and knowledge so that the group can 
get off to a good start. Do not assume that a team really exists and 
expect them to function as a cohesive unit if they haven't worked 
together before. 

To foster a group identity and a deeper understanding of team 
members' roles, you can lead some team-building activities. 
These range from fairly simple exercises to activities which 
require more preparation and resources. Options include: 



Possible Team-Building Activities 

Social activities : Team members get to know each other 
outside the normal work environment. Possibilities include a 
lunch or dinner, a picnic, a reception. 

On-site workshops : Members meet on-site for a workshop to 
cover in more detail the purposes of the evaluation and the 
ground rules they will set for themselves as a team. In so 
doing, they will better get to know each other, air expectations, 
and demonstrate particular skills which the team might use. 

Questionnaires : Members complete a questionnaire prepared 
by the facilitator which nudges them to think about issues they 
will deal with in the evaluation process. (See Activity #2.b.) 
Answers to these questionnaires are summarized by the 
facilitator at the next team meeting. Note, however, that not all 
team members might feel comfortable communicating in 
writing. 

Review of resource materials : Members review a packet of 
basic information about evaluation and discuss it in a meeting, 
workshop, or informally at lunch. Note, however, that not all 
team members might feel comfortable reading materials and 
then discussing them. 
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Discussions with resource persons : Members prepare a set of 
questions they have about team evaluation. As a group, they 
invite others who have had experience in team evaluation or in 
related roles to address their questions. 

Su pport activities for program participants serving on the 
team: Although the spirit of an evaluation team is democratic, 
in practice the roles and hierarchies which govern relationships 
on the floor influence the way team members relate to one 
another in meetings. Education program participants are most 
vulnerable to these potentially harsh dynamics and often need 
to be given special assistance so they can participate actively in 
evaluation teams. Here are four options: 

1. In team meetings, the facilitator encourages worker 
participation by requiring all team members to answer 
certain questions, rather than allowing only a few 
members to dominate the discussion. 

2. If workers are ESOL students, all team members agree to 
encourage those workers to ask what a word or phrase 
means whenever necessary. 

3. Worker members meet on their own (perhaps with 
another trusted team member) in addition to 
participating in regular team meetings. Independent 
meetings give workers a chance to clarify their roles and 
plan out their agendas. 

4. Instructors use class time to help students prepare for the 
roles they are asked to play on the evaluation team. 



ACTIVITY #2.d: 

WITH YOUR NEW MEMBERS : REFLECT ON MEMBERS' 
EXPERIENCE WITH EVALUATION. 

In Step 1, you got yourself into an evaluation mode by 
reflecting on your prior experience with evaluation and thinking 
through the roles you would play in the evaluation process lying 
ahead. Your new team now needs to go through a similar process 
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to orient itself to the notion of collaborative evaluation. Refer to 
your notes from Activity #l.b. as you help your new team reflect 
on their prior experience. 

In a meeting, pose the following questions to the team. 

The program's experience with evaluation so far 

1. What, if any, "evaluation- type" activities have program 
staff carried out so far? 

2. What were the purposes of those activities? 

3. Who set the standards which guided the evaluation? 

4. What did program staff learn from those activities? Who 
used the information? For what purpose? 

5. Based on that experience, what might the team do 
differently in the future when carrying out evaluation 
activities? 

Team members' personal experiences with evaluation 

Experience as an "evaluator" 

1. As individuals, in what situations have you "evaluated" 
someone or something? 

2. In those situations, who set the standards which guided 
your evaluation? Yourself? A group you were part of? Or 
someone or something "outside"? 

3. In those situations, what information did you collect? For 
what purpose? 

4. How was the information used and by whom? 
Experience "being evaluated " 

1. In what situations have you been "evaluated"? (Consider 
the spectrum from getting a report card to a job-related 
performance appraisal.) 
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2. What information was collected about you and for what 
purpose? 

3. Did you help to set the standards you were evaluated by? 
If no: Would the experience have been more meaningful if 
you had helped to set the standards? 

4. Did someone use the information collected about you? For 
what purpose? 

5. How did you feel "being evaluated"? What was 
accomplished? Did you gain anything? What could have 
been done differently? 

Note that time might not permit gathering all team members 
together for a meeting. If so, you might ask those who cannot 
attend to respond to the above questions in a questionnaire or in 
an interview. A meeting, however, is preferred because it helps 
team members to better understand their own and each other's 
perspectives on evaluation, clarify what they have in common, 
and identify who might have particular strengths in the area of 
evaluation. 



In these activities, team members have begun looking forward 
to the evaluation process, laying out possible purposes for conducting 
a team evaluation and members' roles and responsibilities. They also 
looked back at the experiences they and the program have had with 
evaluation. They should now be prepared to work with you to begin 
planning the specifics of their evaluation. 



ACTIVITY #2 e. 

WITH THE TEAM : AGREE ON WHAT NEEDS TO BE DONE TO 
PLAN AN INITIAL ROUND OF EVALUATION ACTIVITIES. 

Team members have now gotten to know each other and have 
familiarized themselves in a general way with the team 
evaluation process. They are about to embark on their first team 
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evaluation and should clarify what they will be doing in the next 
few months. 



Explain that, to help them "get their feet wet" as evaluators, 
they should consider the work they do in the next few months as 
an initial round of evaluation activities. With your guidance, they 
will work as a team to figure out what information they want, 
gather that information, analyze it, and present their findings to 
others. 

In Activity #2.b., you discussed the "why's," "how's," and 
"what's" of the evaluation with potential members. You now have 
agreed cn who will be on the team. If you feel that members still 
need to talk some more about the purposes and responsibilities of 
this first round of evaluation activities, ask each team members to 
respond to the following two questions: 

1. What do you want a team-based evaluation to accomplish? 

2. What would you like your role to be in this evaluation? 

After listening to members' responses, give your own 
perspective on the purposes and responsibilities associated with 
team evaluation. You might explain: 

1. The team will go through a five-phase process to: 

• Identify program goals; 

• Map out the program components needed to achieve those 
goals; 

• Design an evaluation strategy; 

• Gather information from various sources so members can 
decide what is being achieved and what needs to be 
improved; 

• Use the information gathered through the evaluation to 
improve program operations. 
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2. These activities will require time, a willingness to "stretch" 
oneself to try new roles, confidentiality, and a commitment 
to improving the education program. The facilitator is 
available to help all members participate as fully as 
possible, to make this a truly collaborative process. 

3. By working in a team, members can not only help the 
education program but learn how to collaborate around 
other workplace and community issues. 
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PHASE II 



Plan a first round 
of evaluation activities. 



What happens in Phase II 

In Phase II, the team will make the nitty-gritty decisions 
required to plan an initial round of meaningful evaluation activities. 

In Step 3, the team will answer the basic question: " Who wants 
what information for what purpose?" That is, they will map out the 
kinds of questions which team members and other audiences want 
the evaluation to answer, as well as the purposes the stakeholders 
and other audiences will use the information for. 

In Step 4, the team will lay the groundwork for the 
"summative" part of the evaluation which examines to what degree 
program goals have been achieved. Members will clarify the goals 
which stakeholders want the program to achieve, identify 
"indicators" or evidence they might look for to determine if those 
goals are being achieved, and decide which goals they want to focus 
on in the upcoming evaluation. 

In Step 5, the team will prepare for the "formative" part of the 
evaluation in which they examine the effectiveness and needed 
improvements of various program components. Members will think 
through the program components which should be in place to achieve 
program goals. The team will then establish "quality standards" for 
those components and prepare to evaluate the program against those 
standards. 

In Step 6, the team will clarify where they will find the 
information they need, and determine the most effective ways of 
getting that information. 
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Step 3 : Identify who wants what 
information for what purpose. 



You now have a team in place, and the team has agreed in a 
general way what will happen in the evaluation. Now it's time to get 
specific, to plan the details of the evaluation. 

In this Step, the team will respond to the question: "Who wants 
what information for what purpose?" By answering this basic 
question, the team will map out: (1) which audiences, including 
stakeholders on the team, the evaluation should prepare information 
for; (2) the types of information each audience wants; and (3) the 
purposes each audience will use the information for. 

ACTIVITY #3.a. 

ON YOUR OWN AND THEN WITH THE TEAM : CLARIFY WHO 
WANTS WHAT INFORMATION FOR WHAT PURPOSE. 

Before guiding the team through the following exercise, take 
the time to go through each section and think through the 
questions you will be posing to the group. In your evaluation log 
or on flipcharts, draw three columns, label them as indicated 
below, and record your own responses. Clarifying your own 
thinking and anticipating team members' responses will help you 
move the team efficiently through this important step. 

This is especially true in programs where stakeholders have 
distinctly different or conflicting notions about what their 
program should achieve and, therefore, what kinds of information 
an evaluation should collect. Anticipating areas of overlapping 
interests and differences will help you to discuss them with the 
team and prepare a "map" for the evaluation which avoids 
unwanted surprises or conflicts. 

The team's responses to the question "Who wants what 
information for what purpose?" can provide a solid foundation for 
the evaluation. Answering it gives all the team members the 
opportunity to clarify what they feel is important to learn about 
their program, and why. The team members probably represent 
the major audiences the evaluation needs to address. However, 
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you will also have an opportunity to identify other important 
audiences not represented on the team. 

In a meeting, explain that you now want to lay a foundation for 
the evaluation by clarifying (1) which audiences, including team 
members, the evaluation should collect information for; (2) the 
types of information each audience wants; and (3) the purposes 
each audience will use the information for. 

• Title three separate flipcharts "Who," "What Information," 
and "What Purpose." Tape them from left to right across a wall 
where everyone can see them clearly, and where you can write 
on them easily. 

• Ask the team to identify the stakeholders and other 
audiences members might want to prepare this evaluation for. 
List the responses in the "Who" column. 

• Ask the team to select one of the audiences which they want 
to be sure to collect information for. Then ask them to identify 
the questions which that audience would likely want answered 
about the program. Urge team members to be as specific as 
possible. Record the resulting questions in the "What 
Information" column. 

• For that same selected audience, ask team members to 
identify the purpose(s) that audience would likely use the 
identified information for. Record the answers on the third 
sheet titled "For What Purpose." 

As the team identifies the kinds of information various 
audiences want and the purposes the audiences might use the 
information for, members should also consider the format which 
the information would be presented in. For example, if a busy 
funder wants concise measures of program progress, perhaps the 
team will want to present information in numerical, statistical 
formats. On the other hand, some audiences might want "richer" 
information about the program's impact on learners, work teams, 
or the organization as a whole; in such a case, the team might 
prepare in-depth case studies or briefer anecdotes which - using 
the participants' and supervisors' words ~ describe program 
impact. Some audiences might want a mix of statistical and 
narrative information. (Refer to "Key Issues in Design of 
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Information-Gathering Activities" -- Activity #6.b. -- for further 
discussion of the formats you might use depending on your 
audiences' purposes.) 

Once the "What Information" column is filled in, go through the 
responses and, for each type of information noted, jot in 
parentheses whether the information should be presented in 
statistical or narrative format, or some combination of the two. 

• The team has so far identified the questions one audience 
wants answered (along with the formats that information 
might be conveyed in), and the purposes that information 
might be used for. Now ask members to go through the same 
process for each of the other audiences. They may do this 
together as a single group, in smaller groups, or in pairs. 

• If it is helpful, show the team the responses you 
brainstormed on your own at the beginning of this Activity. 

You have now identified a number of possible audiences and 
the kinds of information they might want an evaluation to provide 
them with. You will probably find a pattern among the responses. 
Although they may be phrased in many different ways, 
stakeholders typically want to answer the following key 
questions: 



What stakeholders typically want to know 
in a workplace education evaluation 

"Summative" questions : 

1. What do stakeholders expect tne program to achieve? 

2. What, in fact, is being achieved ~ both in terms of 
stakeholders' expectations and any unanticipated outcomes? 

"Formative" questions : 

3. What are the program's strengths and limitations? What is 
"working" in the program and what is not working so well? 

4. What actions are needed to strengthen the program? 
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ACTIVITY #3.b. 



WITH THE TEAM: SORT YOUR QUESTIONS INTO 
"SUMMATIVE" AND "FORMATIVE" QUESTIONS. 

As shown above, evaluations commonly look for two types of 
information: (1) "summative" (or "outcome "-related) information 
documenting what impact the program is having and (2) 
"formative" (or "process"-related) information focusing on the 
program's strengths and needed improvements. 

Conclude your meeting by asking team members to review the 
list they developed in Activity #3 .a. showing the various kinds of 
information they might collect in the evaluation. As they go 
through that list, team members should indicate which 
information is "summative" in nature (i.e., concerned with 
documenting program goals and outcomes) and which is 
"formative" in nature (i.e., identifying program strengths and 
needed improvements). In a different color felt pen, designate 
each type of information with either a letter "S" (for "summative") 
or "F" (for "formative"). 

Sorting questions in this way will prepare the team for 
subsequent activities where the distinctions between formative 
and summative evaluation are important. 
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Step 4 : Clarify program goals and 
indicators to examine in your 
" summati ve " evaluation . 

Virtually all evaluations want to know to what degree program 
goals were achieved. Most evaluations also would like to know 
whether other, "unanticipated" or "unplanned" results were achieved. 
Put another way, evaluations commonly want the kind of 
"summative" or "outcome" information cited above. Information 
about what is being achieved will also give stakeholders a basis for 
understanding what's working in the program ~ and what's not. 

Before an evaluation team can determine whether 
stakeholders' goals have been achieved, members need to be as clear 
as possible about what those goals are. Clarifying goals is a first step 
toward identifying the criteria against which program impact can be 
measured. 

Ideally, the team should already be clear about what the 
program's goals are. Preferably, members should have been part of 
the program goal-setting process from the start. 

However, the team might now take some time to revisit the 
program's goals for these reasons: 

1. Not all team members might have had opportunities earlier in 
the program to give input into the goal-setting process. They 
might thus not be clear what stakeholders want the program to 
achieve or fully support the goals as they now stand. 

2. New education-related needs and interests might have recently 
emerged in responses to changes in the workforce or 
workplace. Goals should now be revised to reflect those 
changes. 

3. Team members who haven't actively participated in the team 
before will get to know each other better and get practice in 
communicating and making decisions as a unit. 

4. Goal-clarification will help program planners identify clearer 
focal points for instructional activities. 
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ACTIVITY #4.a. 



WITH THE TEAM : MEMBERS CLARIFY PROGRAM GOALS. 

The work which the team has done so far has probably 
indicated that certain stakeholders favor some goals and program 
strategies over others or, at least, that different stakeholders have 
different ways of articulating their goals. In a team meeting, ask 
members to respond to the following questions, to clarify what 
various stakeholder groups want the program to achieve: 

1. What do you think this education program should 
achieve? 

2. More specifically, what should it accomplish for the: 

* Participating workers? 

* The company? 

* The union (where applicable)? 

* Instructors and other education staff? 

3. Are there other stakeholders whose goals should be 
considered? If so, who and what are they? 

Record the goals for each group of stakeholders on a separate 
flipchart. After all members have responded, ask them whether 
they would like to change the list in any way, by adding, 
expanding on, or deleting any particular goals. Then ask the team: 

4. Which goals do you want to focus on most in this 
evaluation? (Which goals do you want to be sure to 
collect information on, so you can know to what degree 
they have been achieved?) 

After you have identified stakeholders' goals, cluster similar 
goals together in meaningful categories. (For example, one set of 
goals might be "job-related," another "personal development.") 
This will help the team to understand the different ways that 
goals might be described by various stakeholders. 

Then prioritize the goals. Decide which goals are relatively 
more important to stakeholders. You might divide the goals into 
"higher-priority" and "lower-priority" groupings. Ranking or 




prioritizing the goals helps the team to distinguish those it wants 
to focus the evaluation on most heavily. 

In such an activity, teams typically generate too many goals to 
examine all at once, especially the first time it undertakes an 
evaluation. Teams should thus try to investigate the goals they 
consider most important. Especially in a first attempt at 
evaluation, s^me teams choose to evaluate goals which are 
relatively easy to evaluate. Here is an example of a completed, 
prioritized list of goals by stakeholder group: 



Sample goals which stakeholders have 
for a workplace education program 



Stakeholders 


Goals 


Higher-level management 


• Employees will improve their oral 
and written communication skills on 
the job. 

• Fmnlovpps will mppt nprsonal oo^ls 
outside the job, including with their 
families. 

• Program will demonstrate how basic 
education might be integrated into 
the company's HRD system. 


Union 


• Employees will bid for better jobs. 

• Employees will benefit in terms of 
personal growth. 

• Employees will participate more 
activelv in the union. 


Supervisors 


• Employees will communicate better 
with supervisors. 


Participants 


• Employees will qualify for 
promotions, improve their job 
security, do their current jobs better, 
advance in their education, and be 
able to help their children with 
schoolwork. 

• Employees will improve oral and 
written communication skills. 


Education staff 


• Other stakeholders will meet their 
goals. 

• Educators will develop their 
expertise in workplace education. 


Outside funder 


• A minimum of 20 employees will 
show improvement in skills and 
knowledge identified as important by 

1 a multi-sector program planning 

1 team. 
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ACTIVITY #4.b. 



WITH THE TEAM: IDENTIFY INDICATORS FOR THE GOALS 
YOU WANT TO EVALUATE. 

Let's say that the team has now identified "improving 
participants' oral and written communication skills" as a high- 
priority goal which it wants to focus the evaluation on. Such a 
goal, while perhaps worthy, is too broad for program planners to 
build specific learning activities around and too vague for the 
evaluation team to measure. The team needs to be much, more 
specific and consciously think through what "improved oral and 
written communication skills" means in this case. Put another 
way, the team needs to identify specific "indicators" for each of 
the goals to be examined. 

An indicator is essentially the evidence, clues, or proof which 
can tell you whether a goal is being achieved. An indicator is 
usually a behavior, attitude, or event that signals the goal is being 
achieved. An indicator "operationalizes" a goal, makes it more 
tangible and measurable. 

To help the team identify indicators, ask them to consider the 
following questions for each high-priority goal they want to focus 
on in the evaluation: 

• How will we know whether this goal is being achieved? 

• What specifically should we look for as evidence or clues? 

• What behaviors, attitudes, and events will "indicate" or signal 
that the goals are being achieved? 

Sometimes this question can be difficult for team members to 
understand. To help clarify, you might give the following 
examples: 
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If the goal is "Program participants will meet personal 
development objectives outside work." some indicators might 
be : 

• Participant successfully enrolls in further education or 

training programs. 

• Participant helps child with school-related tasks: 

~ Understands contents of child's report card. 

- Discusses report card with child's teacher. 

• Participant handles medical-related tasks: 

- Explains medical problems to medical personnel. 

- Understands oral directions from medical personnel. 

If the goal is "Nurses' aides will improve their oral 
communication skills." some indicators might be : 

• Participant can explain daily grooming procedures to 
patient. 

• Participant can participate in staff meetings. 

• Participant can talk with co-workers during break. 

If the goal is "Workers in a manufacturing plant will improve 
their oral and written communication skills." some indicators 
might be : 

• Participant shares ideas about product improvement in 
group meetings. 

• Supervisors understand the participant when the 
participant speaks. 

• Participant uses technical words appropriately in memos. 

If the goal is "Program participants participate actively in 
union affairs." some indicators might be: 

• Participant better understands rights under collective 
bargaining agreement. 

• Participant writes articles for union newsletter. 

• Participant fills out grievance report, if needed. 

• Participant participates actively in union meetings. 

• Participant votes ~ or becomes a candidate ~ in elections 
for shop steward. 

To speed the process, the team might break into smaller group 
to develop indicators for different goals. The small groups could 
then reconvene, share what they came up with, and reach a 



consensus on possible indicators for each priority goal. Goal by 
goal, record the team's suggested indicators on newsprint. 



This is an exercise which requires concentration and creativity. 
Instructors might already have developed a similar list to use as 
objectives for learning activities. If so, instructors might explain 
their list of learning objectives and incorporate that list into this new 
list of "indicators." 

It is unlikely that team members will generate a full list of 
indicators for all goals in one single team meeting. If the team has 
the time, once they have had the chance to practice developing 
indicators for one or two goals, they might agree to "do some 
homework," working as individuals or in pairs to come up with 
additional indicators. They can then re-assemble as an entire team 
to share what they've come up with. Whatever the process used, the 
goal here is to come up with a list of specific indicators which the 
team will look for to know whether stakeholders' various goals are 
being achieved. You will refer to these indicators when you develop 
your data-gathering activities in Steps 6 and 7. 

When trying evaluation for the first time, the team might find 
it difficult to generate a detailed list of goals and indicators. Rather 
than be discouraged, the team should instead start with those goals 
and indicators it feels comfortable with. The team will thereby "get 
its feet wet," gaining evaluation expertise which it can use to refine 
the evaluation process in the future. 
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Step 5 : Identify program components and 
quality standards to examine in your 
" formative" evaluation. 

Until now, the team has been talking mostly about program 
goals and what evidence is needed to determine whether those goals 
are being achieved. In so doing, the team has been laying the 
groundwork for a "summative" evaluation. 

But before moving on to plan details of the summative part of 
its evaluation, it is important for the team to also consider what 
information it might collect for "formative" purposes. That is, the 
team should step back and look in more detail at the other half of the 
"evaluation equation": program "processes" or "components." These 
are all the things that you do in your program that presumably 
enable the program to achieve its goals. These components might 
also be called "strategies" or "operations." 

There are two important reasons for pausing to look at 
program components in more detail at this time: 

1. To ensure that your goals are realistic : In Step 4, you 
clarified your program goals. In order to be sure that those 
goals are achieved, you will need to organize your program 
components (e.g., curriculum, learner recruitment, staff 
training) in appropriate ways. If at this point you realize 
that you will not be able to put in place all the components 
needed to meet the goals you set for yourself, you might 
have to revise your goals to make them more feasible, given 
the resources at your disposal. 

2. To establish a process for monitoring pro gram operations : 
By thinking through at this point what components your 
program needs, you will establish a framework for ongoing 
monitoring ~ and continuous improvement ~ of those 
components. 

The question which the team addresses in this Step is: "What 
do we need to have in place for the program to achieve its goals?" In 
this Step, the team will establish "quality standards" for each 
program component. When the team conducts the "formative" part 
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of its evaluation, it will use these standards to judge the program's 
strengths, limitations, and needed improvements. 

There are two Activities in this Step. It may take one or more 
meetings to complete these activities, depending on the thoroughness 
with which the team defines its program components and quality 
standards. 



ACTIVITY #5.a. 

WITH THE TEAM : IDENTIFY YOUR PROGRAM'S 
COMPONENTS. 

To get started, in a team meeting explain that a workplace 
education program is composed of a number of components. 
These include obvious ones like "instructors," "learners," 
"materials," and "classrooms." They also include less-obvious ones 
like "recruitment process," "staff training," "partnership 
development," and so on. 

• Ask team members to brainstorm the various components ~ 
or "parts" ~ which form their particular education program. 
Ask them to think about the various activities which staff and 
other stakeholders have undertaken to plan and implement 
the program to date. These will probably include various 
needs assessment activities, development of the curriculum, 
recruitment of learners, staff development, and so forth. 

• Record their responses on a flipchart titled "Components of 
the education program: What the team thinks." 

• Explain that researchers have examined workplace education 
programs in the U.S. and Canada and identified an extensive 
list of components which need to be present to ensure that 
programs achieve the kinds of goals discussed in Step 4. 

• Present the following to team members: 
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Vital components of quality 
workplace education programs : 
What research tells us 

1. Goal-clarification : A systematic process for continually 
clarifying all stakeholders' interests, needs, and resources. 

2. Curriculum : A system for designing and implementing a 
learning activities responsive to stakeholders' interests, needs, 
and resources. 

3. Scheduling : A system of scheduling activities appropriate to 
stakeholders' interests and resources. 

4. Staff development : A system of selecting, training, and 
supporting high-quality staff. 

5. Partnership development and maintenance : A system of 
clarifying the responsibilities and roles of all partners (including 
business, labor, education) and facilitating their active 
participation. 

6. On going assessment and evaluation : A systematic way of 
assessing what is being achieved and making needed 
improvements in the program, 

7. Learner recruitment, orientation, placement, and retention : A 
system for recruiting, orienting, placing, and retaining learners 
appropriate to stakeholders' interests and resources. 

8. Facilities : Providing facilities which support (rather than 
present an obstacle to) learning. 

9. Fiscal management : A fiscal management system to ensure that 
all program components are adequately supported. 

• Ask the team to compare their own list of possible program 
components (on the flipchart) with the above list of 
components suggested by research. 
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• Ask the team to now decide which of the components on the 
two lists they consider to be key components of their own 
education program. Agree on a list of components necessary 
for your program. 

The resulting list will be general in nature, outlining a list of 
things which need to be in place to ensure an effective program. 
The team now needs to be more specific and decide what high- 
quality components will look like, given the program's particular 
goals and circumstances. 

ACTIVITY #5.b. 

WITH THE TEAM : AGREE ON QUALITY STANDARDS FOR 
EACH COMPONENT. 

Once they have identified necessary program components, the 
team is ready to map out "quality standards" for each component. 
Quality standards are the standards which each component of the 
program should maintain in order to ensure that the program 
meet its various goals. Think of quality standards as the "parts" 
needed to make a component work properly. 

For example, take component #6: "Learner recruitment, 
orientation, placement, and retention." In an ESOL program, 
recruitment is generally conducted in the native languages of the 
potential employees. In such a case, team members may specify 
that having multi-lingual recruitment strategies is a "quality 
standard." 

In a workplace basic reading and writing program, where 
employees may feel embarrassed that they don't know how to 
read very well, confidentiality of recruitment will likely be very 
important. Team members might thus specify that confidential 
recruitment is a "quality standard." 

It is up to each team to specify the quality standards for each 
of their program components. Before asking the team to do this, 
you might model how you would establish quality standards for 
one of the program components. For example, what are quality 
standards for the component, "staff development"? That is, what 
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must be in place in the "staff development" component of the 
program to ensure that the program meets its various goals? 



To do this, lead the team through the following process: 

1, Prepare a grid with the program goals across the top and 
your program components down the left side, as shown in 
the following model: 

Goals and components : 
A grid for identifying quality standards 



Learners 

improve 

job-related 

oral and 

written 

skills. 



Learners 
are able to 
use basic 
communi- 
cation skills 
for tasks 
outside the 
job. 



Program 
shows how 
basic ed. 
can fit into 
company 
human 
resource 
dev. system 



Learners 
qualify for 
job promo- 
tions. 



Learners 
participate 
more 
actively in 
the union. 



Goal clarifi- 

cation 

Curriculum 

design & 

imple- 

mentation 

Scheduling 



Staff 

dev f ment. 
Partner- 
ship dev. & 
main- 
tenance 
Ongoing 
assess- 
ment & 
evaluation 
Learner 
recruitment, 
placement, 
orientation, 
& retention 
Facilities 



Fiscal 
mgt. 
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2. To develop a set of quality standards for the staff 
development component, proceed one-by-one through each 
of the boxes on the grid to the right of "staff development." 
In each box, consider what staff-development activities you 
need to have in place to ensure that the corresponding goal 
(at the top of the column) is achieved. 

For example, to help ensure that the first goal of 
"Learners improve job-related oral and written skills" is 
achieved, staff development should include the following: 

What needs to be in place 
in the staff development component 
to ensure that 
"Learners improve job-related oral and written skills"? 

• Instructors are trained in how to assess learner job- 
related communication skill needs and interests. 

• Instructors are paid and given adequate time to assess 
needs and develop appropriate curricula. 

• Staff meet once every two weeks for an hour and a half 
to discuss their work. 

• Staff are well-represented on the evaluation team. 

• Staff participate in hiring of new staff. 

• Staff are encouraged and helped to attend local, 
regional and national conferences on workplace 
education. 

3. When you have completed all the boxes for "staff 
development," present your ideas for possible quality 
standards for staff development to the team. Ask them to 
refine and expand on your list. 

4. Now ask the team ~ as a large group or in smaller groups - 
to do the same for all xba other program components. That 
is, they should map out a set of "quality standards" - 
things which need to be in place ~ which will enable each 
program component to contribute to the successful 
achievement of each of the program goals. 
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This linking of program components to program goals might be 
a familiar concept for those already familiar with the 
fundamentals of quality control, in which "operations" are linked 
to "goals." Linking the "what" in what you want to achieve back to 
the question of "how" you achieve it is a basic building block of 
quality management. However, if team members are not familiar 
with this approach, it may seem a little awkward. Take as long as 
you need to show the value of linking program components to 
program goals. 
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Step 6 : Decide how you will collect the 
information you need. 

The team has now established a foundation for the formative 
and summative parts of their evaluation. To carry out a "summative" 
evaluation (that is, to document program impact or outcomes), the 
team will examine the "indicators" it generated in Step 4. To conduct 
a "formative" evaluation, the team will examine whether the various 
program components are meeting the "quality standards" it 
identified in Step 5. 

Keep in mind that the boundaries between "summative" and 
"formative" evaluations should not be seen as rigid. In fact, a 
"formative" evaluation depends on what is found in the "summative" 
side of the evaluation. For example, for a formative evaluation to 
determine whether the content of the curriculum is relevant, it will 
be useful to know what the summative evaluation has found 
participants to be achieving using that curriculum. As noted earlier, 
a comprehensive evaluation normally examines both "summative" 
and "formative" questions. 

In this Step 6, the team will decide where and how it will 
collect the information it needs for "summative" and "formative" 
purposes. 

Where will the team get the information it needs to know 
whether goals are being reached? And where will members get 
information to clarify the strengths, limitations, and needed 
improvements of various program components? To what degree can 
the team rely on itself as a source of information, and to what degree 
will they look to sources outside the team? 

How will the team get information from those sources? Will 
members use team meetings to brainstorm what they already know 
about the program? Will they use "tests," questionnaires, 
interviews, or focus groups to get information from sources outside 
the team? Will members examine production or classroom records? 
These are the kinds of questions you and the team will now answer. 
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ACTIVITY #6.a. 



O N YOUR OWN : THINK ABOUT WHERE AND HOW YOU 
MIGHT COLLECT THE INFORMATION YOU NEED. 

To answer the question of how you will collect the information 
you want, you might first work on your own. As with other Steps, 
preparation on your own will facilitate work with the team. 

As you turn to the questions of "where" and "how" you will 
collect the summative and formative information you want, keep 
in mind that there is a wide range of possible information sources 
and information-gathering activities you might choose from. Take 
some time now to consider which sources and activities might 
produce the kinds of information you want for summative and 
formative purposes. 

"Summative" activities 

In Activity#4.b., you identified "indicators" or "evidence" which 
can tell you to what degree and in what ways the program is 
making progress toward its various goals. Now you need to figure 
out where and how you might collect that evidence. 

Start by reviewing the lists of indicators which the team 
developed in Activity #4.b. As you review that list, ask yourself 
what sources you might go to to find those indicators. Remember 
that those indicators are evidence which can inform you to what 
degree program goals are being met. Jot down your list of 
possible information sources. 

You are likely to identify a number of possible sources. These 
include human sources like team members themselves, as well as 
other learners, managers, union representatives, and instructors. 
You might also have identified material sources like class records 
or company or union documents. For example, material sources 
may include: in-class materials; attendance logs and sign-in 
sheets; personnel records; safety records; union meeting 
attendance records; and records of bids for better jobs. 
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Then ask yourself how - by what means ~ you might get the 
information you want from each of those sources. Ways of 
collecting information might include: 



Interviews : The team might conduct individual 
interviews with learners, instructors, supervisors, union 
representatives, or others. These informants might be 
asked whether they have noted any impact from the 
education program. The informants might, in particular, 
be asked whether they have observed the particular 
"indicators" of program impact you developed in Step 4. 

Focus groups : The team might conduct focus groups 
(group interviews) with learners, instructors, supervisors, 
union representatives, or others. You might ask 
essentially the same questions referred to under 
"Interviews" above. 

Questionnaires and checklists : The team might ask 
learners and others to respond in writing to 
questionnaires and checklists prepared by the team. You 
might ask essentially the same questions referred to 
under "Interviews" above. 

Tests and simulations : Workplace educators often 
question the relevance of standardized tests. 
Nonetheless, such tests are often used in workplace basic 
skills programs to provide some kind of evidence of 
"learner ability." Note that many practitioners also 
develop their own "tests," focusing on the learning 
objectives which they develop with input from learners 
and other stakeholders. In some cases, these customized 
"tests" take the form of "simulations" in which learners 
demonstrate in role plays (mock situations) what they 
can do. Instructors or learners themselves then rate the 
learners' "performance" as a way of documenting learner 
progress and needs. These various kinds of tests can 
produce evidence of program impact of possible use to 
the team. 
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Review of classroom records : The team might review 
evidence which instructors and learners are already 
collecting as part of the instructional process. For 
example: 

• Instructors and learners might be rating learners' 
abilities to perform certain tasks (i.e., 
"competencies") based on learners' performance of 
those tasks in classroom simulations or in "real" 
situations outside the classroom. 

• As noted under "Tests and simulations" above, 
instructors might have been using a particular 
standardized test or might have developed one 
specifically for this program. The team might review 
any existing records from such "tests." 

• Learners might be collecting sample writing, lists of 
materials read, or other work in "portfolios." Such 
materials serve as tangible, "authentic" evidence of 
what learners are doing. 

• Classes might be videotaping or audiotaping 
classroom activities. The team might review such 
tapes as evidence of learner progress. 

Review of program "log s": Team members and others 
might record observations, suggestions, and other 
relevant information about the program in various kinds 
of "logs." One version of this is "individual" logs which 
individual team members keep handy to record 
observations, hallway conversations, questions, and ideas 
related to the program. This would allow them to capture 
anecdotes or other telling evidence of program impact, 
program strengths, and needed improvements. Members 
can then share the content of these journals in regular 
team meetings and in periodic program evaluations. 
Alternatively, team members and anyone else with 
knowledge of the program might be encouraged to record 
such information in a "collective" log kept in the program 
office or other convenient location. 



Review of documents from meetings and other events : 
The team might previously have met and kept records of 
those meetings. What information might be contained in 
those records which can be used for evaluation purposes? 

Review of company or union records : Company and 
union records may provide information which will 
supplement information gathered from other sources. (As 
with all records, these should be assessed critically for 
inaccuracies.) In addition to formal "records," the 
company or union might supply (with learners' 
permission) samples of writing which learners have done 
(e.g., memos, work orders, newsletter articles) for the 
evaluation team to review. 

Observation of classroom activities : Members might 
observe classroom activities, to identify learner progress 
and see how such components as the curriculum, 
facilities, etc. are working. 

Observation of learners using skills outside the classroom : 
Members might observe participants at work, in union 
activities, and/or in extracurricular activities (like 
recognition ceremonies) applying the knowledge and 
skills they developed in the classroom. 



As you think through possible information sources and 
information-gathering activities, you might record your responses 
on a matrix or grid. Down the left side, list possible sources of 
information. Across the top, record the possible ways of collecting 
the information you want. Each box in. the grid, then, will 
represent a possible way of collecting information from a 
particular source. 

Here is a sample grid: 



60 

53 



Possible information sources 
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Once you have brainstormed possible sources and activities for 
collecting information, go back to the grid and mark with an "S" 
those source/activity pairs which you feel might be most 
appropriate to help you answer the "summative" questions. 
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You have now mapped out possible ways of getting 
"summative" information, information which will help you 
document ("summarize") (1) to what degree and in what ways the 
program has met its goals and (2) other evidence of program 
impact. 

"Formative" activities 

The sources and activities you use to collect information for the 
"formative" side of your evaluation might differ somewhat from 
those you use for your "summative" evaluation. In many cases, 
team members might be the best source of specific information 
about various program components, because they have been 
intimately involved in planning and running those operations. 
That knowledge might be tapped relatively quickly in a team 
meeting or in individual interviews with one or two key team 
members. 

In Activity #5 .a., the team identified the components necessary 
for successful implementation of your program. Then, in Activity 
#5.b., you identified the "quality standards" for each of those 
components. The "formative" part of your evaluation should 
answer these two questions: 

Questions to answer 
in the "formative" part of your evaluation 

1. To what degree are your various program components 
meeting the quality standards you have set? 

2. What actions are needed to ensure that program components 
meet your quality standards? 

With this in mind, look at the grid of possible information 
sources and information-gathering activities you developed 
earlier in this Activity for the "summative" part of your 
evaluation. Ask yourself which of those sources and activities you 
might use to answer the above two "formative" questions. Mark 
the corresponding boxes on the grid with the letter "F." 

You have now developed a grid of possible information sources 
and information-gathering activities, and then designated which 
sources and activities you might use to answer your "summative" 
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and "formative" questions. By so doing, you should now have a good 
understanding of the various information-gathering activities the 
team might use in its first round of evaluation activities. 



ACTIVITY #6.b. 

ON YOUR OWN : CONSIDER KEY ISSUES IN DESIGNING OF 
INFORMATION- GATHERING ACTIVITIES. 

Before going further in discussing possible information- 
gathering activities with team members, it is important for you to 
be prepared to deal with some key issues in the collecting of 
information. Consider the following: 

Key Issues 
in 

Design of Information-Gathering Activities 

Confidentiality and anonymity 

All team members should understand the need to respect 
the confidentiality of what is said within the team and in 
communications with others outside the team. The challenge is 
to ensure trust and open dialogue among all involved in the 
evaluation process, in the spirit of collaboration and continuous 
improvement. Any sense that the evaluation process is a 
threat should be avoided. 

If potential sources of information feel threatened by 
telling the truth about their experiences, they will probably not 
reply frankly. There are two ways of reducing threat and 
encouraging open dialogue: 

• In all team activities, establish clear groundrules that the 
sources of anything said will not be repeated to anyone else. 
When talking about what was said, don't cite sources by 
name. 
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• If necessary, collect information anonymously, so that no 
one ever knows the source of a j articular piece of 
information. For example, sources might complete 
questionnaires and submit them to the team without signing 
them. Or you might provide a suggestion box in which 
learners or others can drop comments about the program. 

Where a spirit of trust and dialogue has been established, 
information sources might be eager to express their opinions 
and happy to sign their names to questionnaires or interviews. 
The team will have to judge the relative merits of anonymity 
versus disclosure. 

Unobtrusiveness 

While a team might want to collect as much information 
as possible, it must be careful in its data-gathering not to be 
unreasonably obtrusive. For example, some data-gathering 
activities might require too much time of either the 
information source or those gathering the information. Some 
activities might be overwhelming in terms of the number of 
questions being asked. If you ask for too much, your 
respondents may refuse to give you anything at all. 

A related issue is the cultural appropriateness of your 
data-gathering activities. If people are accustomed to working 
in groups and discussing their work openly, then focus groups 
might be a "culturally acceptable" way to collect information. 
If employees are accustomed to filling out check-lists and 
questionnaires, these methods of data collection might offer a 
comfortable way to start data collection. 

Subtleties of how communication is normally carried out 
should also be paid attention to. For example, some employees 
might not like being interviewed by someone of a different 
gender. Some might feel it is taboo to say anything critical 
about a respected figure like a teacher. Or some might see a 
workplace education program as a rare opportunity and not 
want to jeopardize it by criticizing it in any way. 
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Validity 



To be valid, an information-gathering activity must 
actually measure what we want it to measure. For example, a 
test of students' learning in a particular course must cover a 
balanced representation of the content and skill level actually 
dealt with in the course. Otherwise, any resulting scores may 
present a misleading story about whether the students have 
mastered the course material. 

Similarly, imagine that an employee is asked ~ as part of 
an evaluation - to read a comp?my sign that says* "Caution! • 
Wear safety glasses." The sign is illustrated with a large 
picture of safety glasses. The employee reports that the sign 
says: "Wear safety glasses." But is the employee reading the 
words or the picture?. The validity of the item is questionable. 
(See below, "Field testing your instruments.") 

Reliability 

To be reliable, an information-gathering activity must be 
accurate and consistent in what it measures. For example, if 
the same informant responds in significantly different ways 
when asked the same question twice in a short time period, 
you might have to reconsider the question's reliability. Also, if 
two different interviewers ask the same question of the same 
program participant, they should get similar information. (See 
below, "Field testing your instruments.") 

Triangulation 

Triangulation means that, rather than rely on only one 
source of information or one method of gathering information,' 
you gather information from several sources and perhaps use 
several different methods of gathering it. For example, you 
might ask essentially the same questions of learners, 
supervisors, and instructors, rather than rely only on what 
instructors say. Triangulation gives you a more rounded set of 
information to base your judgments on. It gives you the 
opportunity to "check and balance" the perceptions of one 
group against another. 
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Imagine that you ask supervisors and program 
participants to rate the participants' improvement in 
communication skills on a questionnaire. If supervisors rate 
the amount of improvement significantly lower than the 
employees themselves, the evaluation team can ask: why does 
the information differ so much from source to source? This 
kind of question helps the evaluation come closer to 
deteiTnining the actual status of participants' communication 
skills. 

Experimental design 

Often, when people think of research and evaluation, they 
think of a design that compares two groups of people. One 
group (termed the "experimental" group) receives the 
"intervention" — the special service ~ that the research is 
analyzing. The other group does not receive the service and is 
considered the "control" group. The control group is usually 
matched as closely as possible to the "experimental" group. If 
receiving the special service is the main difference between the 
two groups, then differences that become apparent in the 
experimental group are understood to be the result of receiving 
the special service. 

There is value in evaluating your program with a control 
group. Ideally, it tells you more clearly what your program - 
and only your program ~ is achieving. If you are interested in 
conducting your evaluation with a control group, refer to the 
resource materials in the Guide's bibliography to learn more 
about experimental design. 

Keep in mind, however, that producing such a design is 
difficult and perhaps beyond the means of a typical workplace 
education program. For example, you might have to develop 
specific technical skills or hire a consultant who already has 
them. Note, too, that identifying two groups with similar 
characteristics within the context of a changing workplace 
might be very difficult or impossible. 

Most internal evaluations of workplace basic skills 
programs do not employ control groups. The information 
which these evaluations produce can nonetheless be highly 
valuable. An evaluation which provides richly descriptive 
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information about program goals and processes, especially 
information collected over time, can enable you to improve 
your program as readily as results from an experimental 
design. 

Other options which are more-readily available to 
internal workplace education program evaluations include: 
comparing the achievements of different groups of participants 
(e.g., this year's learners vs. last year's participants) over time 
and following the achievements of individual participants over 
time. 

Monitoring outcomes over time 

Whether or not you use a control group or compare 
different groups of participants, there is a value in 
documenting the progress of your program participants over 
time. By so doing, you will in effect be "comparing them to 
themselves." You can compare their knowledge, interests, and 
behaviors prior to starting the program with those same factors 
later in the program and perhaps after the program is over. 

To do so, you will have to collect information from 
participants either prior to or in the early stages of their 
participation in the program - at "baseline." Good baseline 
information is vital if you are to be able to monitor progress. It 
is worth the extra effort that might be needed to collect it. 

Recognizing that many potential participants might be 
scared off by "tests" - especially those conducted by 
instructors they don't even know ~ some practitioners have 
developed less-threatening and possibly more-accurate ways 
of assessing learner abilities and interests at baseline and later. 
These include interviews and other exercises in which potential 
participants identify and demonstrate what they are interested 
in learning and how they already use literacy. The resulting 
evidence is compiled in a learner portfolio. It is used by the 
instructor to plan activities relevant to learner needs and 
interests and it also serves as baseline information. 
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Qualitative vs. quantitative information 



Generally speaking, qualitative information is 
characterized by narrative description and interpretation, and 
quantitative information is compiled and analyzed numerically. 

Although many tend to think of these kinds of 
information as mutually exclusive and having different value 
(even their slang variations - "soft" and "hard" - indicate 
value), neither premise is accurate. Each type of information 
can be useful, depending on the functions it performs and the 
audiences it serves. It is also possible ~ and often desirable ~ 
to render qualitative information in quantitative terms in order 
to clarify its meaning. In this way, two ways of documenting 
the program enrich rather than compete with one another. 

For example, assume that you have conducted interviews 
with ten supervisors. You asked the supervisors a general 
question about what they think the results of the education 
program are. The answers the supervisors gave you are in 
sentences and the content varies, although there is some 
overlap in what supervisors describe as important results: 
improvements in self-esteem and communication skills. You 
value the richly descriptive information that your interviews 
have given you, but you are also interested in knowing how 
many supervisors observe improvements in self esteem and 
communication skills. So you count how many responses you 
have for each category. 

Seven out of ten supervisors say that improved self- 
esteem of employees is a result of participation in the 
workplace basic skills program. Six out of ten supervisors cite 
improved communication skills as a result. By presenting both 
a narrative description of what the supervisors said and some 
simple numbers about the frequency of key responses, you can 
maintain the rich descriptions resulting from open-ended 
interviews while providing a useful numerical summary of 
major themes. 

Although quantitative information is not intrinsically 
more valuable than information presented in qualitative form, 
nor vice versa, some team members and other audiences may 
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have expectations that a "real" evaluation will provide 
quantitative evidence that the program is working. Each team 
must assess the kinds of information that its audiences will 
consider valuable. The team may either provide that 
information or challenge its audiences' tMnking about how 
evaluation results are best presented. 

It is important to note that "soft" information-gathering 
methods borrowed from anthropology, sociology, and 
marketing are becoming increasingly popular in workplace 
education evaluation because of the depth and detail of 
information they provide. These "soft" methods include careful 
documentation of meetings, unobtrusive observations of work, 
focus groups, and program logs or portfolios which are kept 
collectively by program stakeholders. As workplace education 
programs focus more on supporting quality management and 
helping to bring about organizational change, these "soft" 
methods become more appropriate. They fit with a more- 
thoughtful, participatory management approach that is 
interested in involving all employees in continuous feedback 
and improvement. 

Statistical precision of information 

An issue related to the merits of qualitative vs. 
quantitative information is the need for statistically-precise 
analyses of information. If you decide that your evaluation 
will present its findings primarily in quantitative or numerical 
terms, you will be faced with the question of how sophisticated 
you want or need those numbers to be. Even if you keep your 
numbers simple - reporting of frequencies of responses and 
"cross-tabulations" matching types of responses to particular 
information sources- you may find it very helpful to use a 
computer database to organize your information. There are 
many database programs which can help you with simple 
statistical analyses. Most personnel and training offices use 
one. It might be possible to "borrow" it for the evaluation. 

If numerically-rendered information is important to your 
evaluation, developing a database which contains the 
"numbers" generated by your evaluation over time will be very 
valuable. If no one on the team has the skills needed to 
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oversee the development of the database, you should find 
someone else inside -- or outside - your program to help you. 



You will have to be the judge of whether all your team 
members will want to discuss these issues. For some, many of 
these issues might seem like technicalities which only the 
facilitator needs to be concerned about. When these issues come 
up in your discussions with team members, you might briefly go 
over these issues, perhaps providing a handout based on the 
information presented above. Explain that some of the issues ~ 
especially confidentiality, obtrusiveness, and the type of 
information that your audiences consider valuable - are things 
which all team members need to be concerned about. Clarify your 
own position on these issues and get the team's feedback. 



■ ACTIVITY #6.c. 

fl WITH THE TEAM : DECIDE WHICH ACTIVITIES TO USE TO 

COLLECT YOUR INFORMATION. 

■ When you meet with the team next, explain that -- in Steps 4 
and 5 - they identified two kinds of information they might 

I collect: "summative" information to determine whether goals are 

being met and "formative" information about the strengths and 
— needed improvements of various program components. Explain 

| that they now should take the next step of deciding where (the 

they will collect that needed information and how they will get 

■ the information from those sources. 

To do so, lead them through a process similar to the one you 

■ personally went through in Activity #6.a., as follows: 

. "Summative" activities 

Show the team the list of "indicators" they developed in 
fl Activity #4.b. Explain that these indicators are essentially the 

• "evidence" which can tell you to what degree and in what ways 

the program is making progress toward its various goals. Explain 
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that you now need to figure out where and how you might collect 
that evidence. 



As members review the list of indicators, ask them what 
sources the team might go to to find those indicators. Jot down 
their responses on a flipchart titled "Possible information sources." 
Members are likely to identify a number of possible human and 
material sources, similar to the sources you personally developed 
in Activity #6.a. 

Then ask members how -- by what means -- the team might 
get the evidence (indicators) of progress they want from each of 
those sources. Record their responses on a flipchart titled 
"Possible ways of collecting information." 

Give members time to think and don't be alarmed if they don't 
come up with as extensive a list as you did in Activity #6.a. (But 
don't be too surprised if they develop an even more complete list 
or think of ways of getting information you hadn't imagined!) 

When the team has developed its lists of possible information 
sources and information-gathering activities, explain that you 
have already developed similar lists (in Activity #6.a.) and laid 
them out in a "grid" format. Show them your grid of "Possible 
information sources and information-gathering activities." 

Compare your grid with the team's two lists of sources and 
activities. On a new flipchart, you might develop a revised grid 
which all agree are possible sources and activities you might rely 
on to gather evidence (indicators) of progress toward program 
goals. As you did personally in Activity #6.a., you might mark the 
appropriate boxes on this new grid with the letter "S" to designate 
that this is a source/activity pair you might rely on for the 
"summative" part of your evaluation. 

"Formative" activities 

Explain to the team that they have now identified possible 
ways of collecting the information ("evidence," "indicators") they 
need for the "summative" part of their evaluation. Explain that 
they can now go through a similar process to identify where and 



how they might get information for their "formative" evaluation 
activities. 

Tell the team that the sources and activities they use to collect 
information for the "formative" side of your evaluation might 
differ somewhat from those they use for their "summative" 
evaluation. In many cases, team members might be the best 
source of specific information about various program components, 
because they have been intimately involved in planning and 
running those operations. That knowledge might be tapped 
relatively quickly in a team meeting or in individual interviews 
with one or two key team members. 

Point out that, in Activity #5 .a., the team identified the 
components necessary for successful implementation of your 
program. Then, in Activity #5.b., members identified the "quality 
standards" for each of those components. 

Ask them to now go through the list of quality standards they 
developed for each program component in Activity #5.b. As they 
consider each set of quality standards, ask members to identify 
where and how they might get the information they need to 
answer the following questions: 

Questions to answer 
in the "formative" part of your evaluation 

1. To what degree are your various program components 
meeting the quality standards you have set? 

2. What actions are needed to ensure that program components 
meet your quality standards? 

Because the team probably includes several people who have 
an intimate knowledge of the various program components, take 
some time to pose the above two questions to the team. For each 
component, record members' responses on a flipchart titled "Team 
members' assessment of various program components." 

By going through such an exercise, team members will generate 
useful insights about the program components, based on their own 
involvement with those components. Members will also better 
understand the mechanics of asking "formative" evaluation 
questions. ^ 0 
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When members have themselves analyzed to what degree 
various components are meeting quality standards, ask them 
whether they still need to go to other sources to confirm or 
expand on their own analysis. If so, agree on what information 
sources and information-gathering activities to use to collect that 
additional formative information. As they identify those sources 
and activities, mark the corresponding boxes on the grid with an 
"F." 

Note that, as members review their earlier lists of "quality 
standards," they might find that they need to revise the lists. 
Encourage them to do so, so that their lists of quality standards 
are as up-to-date and complete as possible. 

By the end of this Activity, the team should have a grid which 
indicates where and how they can get the "summative" 
information they need to assess program impact to date and 
"formative" information to clarify whether program components 
are meeting the team's "quality standards." 

The team is now ready to begin the next Phase. In Step 7, 
members will construct a customized information-gathering 
"instrument" or procedure to collect the information it wants. 
Then, in Step 8, members will field-test that instrument and apply 
what they learned to developing additional instruments. 
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PHASE III 



Prepare to gather 
your information. 

What happens in Phase III 

In Phase III, the team will prepare to collect the information it 
wants. 

In Step 7, the team will design the activities it needs to collect 
that information. 

In Step 8, the team will field- test their activities for gathering 
information. 

In Step 9, the team will map out a strategy - an action plan — 
for collecting and analyzing the information it wants. 



74 

67 



Step 7 : Design your information- 
gathering activities. 



The process of designing customized activities for gathering 
information is labor-intensive but need not be overwhelming. It 
requires one or more individuals with time, creativity, and interest. 
The team can decide how far it wants to go. 

• How many information-gathering activities will it design? 

• What types of questions ("summative" and/or "formative") 

will each activity focus on? 

• How many questions will each activity include? 

• Will questions be structured to produce "quantitative" 

information ("statistics," "numbers") or more-" qualitative" 
information (in narrative form)? 

Your answers to these questions will depend on the kinds of 
information you want your evaluation to produce to satisfy your 
audiences. 

In some cases, the team facilitator might be the only one 
available to do much of the design work. If this is true, formal or 
informal collaboration with one or more colleagues from the local 
network of adult educators might make the work more manageable. 

In other cases, team members will be available to design and 
field-test their information-gathering activities. Some team 
members may even have experience with this kind of work. The 
team might form a subcommittee for the design of its information- 
gathering activities. 

The team will have to decide who will be responsible for 
putting together drafts of "instruments" (i.e., the plan for each 
information-gathering activity) to present to the other team 
members. As you develop your activities, remember that few people 
have had experience developing information-gathering instruments 
specifically for a workplace education context. The team should 
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acknowledge that and see this stage as an opportunity to "get their 
feet wet" to develop both draft instruments and their own expertise. 

See this as an experimental phase in which the team can 
develop their instruments, field-test them, and refine them again. 
Refer to the bibliography at the end of the Guide for additional 
resources on design and field-testing of evaluation instruments. 



ACTIVITY #7.a. 

ON YOUR OWN AND/OR WITH THE TEAM : PREPARE A 
DRAFT OF ONE INSTRUMENT. 

An "instrument" (or "tool," "procedure," "schedule," or 
"protocol") is, in effect, a script which an evaluation team can use 
to collect the information it wants. In Steps 4 and 5, the team 
agreed on the summative and formative information it needs. In 
Step 6, members identified sources for that information and 
activities members might use to get that information from those 
sources. 

As shown in Step 6, there are many options for collecting 
information. In this Step, we will now focus on developing a 
questionnaire which will be completed by participants, union 
stewards, and supervisors in slightly different formats. We 
choose a questionnaire because it is a standard evaluation tool, 
and the issues which you confront in constructing one are 
virtually the same that you will confront if you conduct 
interviews or focus groups - two other common means of 
collecting information. Working through these issues in the 
questionnaire, interview, and focus group formats will also help 
you to clarify what to consider in other information-gathering 
procedures. 

Here are some guidelines for developing a questionnaire or any 
other instrument. 
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Adapt your core questions for use with different 
sources and in various information-gathering 
activities. 

In Step 4, the team identified indicators with which to 
measure progress toward goals in the "summative" part of 
the evaluation. In Step 5, members identified quality 
standards to measure the effectiveness of various program 
components in the "formative" part of the evaluation. In 
Step 6, the team thought through what kinds of information- 
gathering activities it might use to collect information related 
to those indicators and quality measures. 

The team should now transform its lists of indicators and 
quality standards into an interrogative ("question") format. 
For example, if one indicator is "Participants speak up with 
useful suggestions in committee meetings," it might be 
converted to "Do participants speak up with useful 
suggestions in committee meetings?" 

By converting the indicators and quality standards 
established in Steps 4 and 5 into a question format, the team 
will have a set of "core questions" which members can now 
use in a questionnaire and ~ with modification - again in 
other information-collection activities. 

Use plain language. 

Do not confuse the person completing the questionnaire 
by using fancy or technical language. If the language you are 
using is English, use simple, direct, common English. Keep the 
tone friendly. This is especially relevant if the intended 
respondent is a new speaker of English. Similarly, make sure 
that the directions you give about how to answer questions 
are clear. (The team will have to decide whether to use 
languages other than English when communicating with 
speakers of other languages.) 

Ask for all necessary demographic information. 

Demographic information enables you to describe those 
who answered your questionnaire. It generally includes age, 
gender, race/ethnicity, length of time spent in the 
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organization, job title, and the course enrolled in. Such 
information enables you to link your results back to the 
types of people who give you information. 

For example, if roughly half of those responding to your 
questionnaire say that they always speak up in meetings and 
roughly half say that they do not speak up in meetings, it 
might be important to see who does/does not speak up. If it 
is mostly women who do not speak up, then you may have 
learned something important. In response, you might 
recommend that the company determine whether particular 
conditions in the company culture make it difficult for 
women to speak up in meetings. You might decide to set up 
special activities to enable women to participate more 
actively. 

To maintain confidentiality, code each questionnaire. 

If necessary, use a number or other identifying mark on 
the questionnaire to enable you to keep track of who is 
responding while keeping their identities confidential. 

Limit the number of questions you ask. 

Carefully consider the number of questions you ask. 
Consider how much time your respondents will have to give. 
Think through how much time it will take you to analyze 
answers. A handful of well-chosen and well-worded 
questions is worth more than pages of questions which will 
wear out both the respondent and the evaluator. Remember 
that for every extra question you ask, you will have to 
analyze responses to that question multiplied by the total 
number of respondents. If you are going to conduct a survey 
with 50 participants, for example, you will have fifty 
answers to analyze for every question you ask. 

State only one question at a time. 

It is easy to make the mistake of combining two 
questions in one. If, for example, you want to know whether 
participants are improving in their oral and written 
communication skills, you might be tempted to ask a 
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question like: "When your supervisor speaks to you, do you 
listen carefully and understand her?" 

In this question, you are asking the respondent whether 
(1) s/he listens carefully and (2) s/he understands. These 
are two separate questions. If you join them in one question, 
you will not know whether your respondent is referring to 
the "listening" question or "understanding" question. The 
responses thus won't be of much help. 

For a numbered scale, use only an even number of 
responses. 

Questionnaires often use scales to pre-code answers 
numerically. For example, many of us have received 
marketing surveys in the mail which ask us to rate our 
satisfaction with a product on a scale of one to five, with 
"one" designating "not at all satisfied" and "five" being "very 
satisfied." 

If you want to use a scale to quantify some or all of the 
possible responses in your survey, make sure that you use 
an even-numbered scale -- not an odd-numbered scale like 
the one in the marketing survey. This is because odd- 
numbered scales give the respondent the option of taking 
the easy road "down the middle." On a five-point scale, s/he 
can always choose number 3, and never have to make 
sometimes uncomfortable, discriminating decisions about 
what s/he really thinks. An even-numbered scale forces a 
decision that is closer to one end of the scale or the other. 

Provide an adequate number of possible responses. 

When using a numbered scale, there are times when you 
need to provide more than just a few possible answers. Do 
this to allow your respondents to change their answers if 
you ask them to respond to the same scale two or more 
times over a period of time. For example, rather than giving 
respondents only four choices ("poor," "fair," "good," and 
"excellent"), you might give them six ("totally unacceptable," 
"not good," "mediocre," "fair," "good," "excellent"). This would 
allow them to change their responses over time, to 
demonstrate a change in their thinking on the question 
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asked. With too few choices, you are unlikely to be able to 
see any change, even if the respondents do change their 
perspective on a particular question. 



Note, however, that you shouldn't go too far and provide 
too many possible responses, because (1) it can be confusing 
to the respondent to have to "split hairs" among very- 
similar categories, and (2) the differences among categories 
become so slight that you will end up combining categories 
in your analysis, anyway. 

Decide whether you wili use open-ended or closed- 
ended questions — or a combination of the two. 

Generally speaking, questionnaires ask either "open- 
ended" or "close-ended" questions. As the name indicates, 
open-ended questions do not specify possible answers. They 
allow respondents to answer in their own words without too 
much prompting. For example, an open-ended question 
might ask: "What has been the impact of the class for you in 
terms of improving your communication skills on the job?" 

There are some advantages to using open-ended 
questions, especially when you are not sure what you are 
looking for and need to explore the full range of possible 
participant answers. Open-ended questions tend to make 
people "think" about rather than merely "respond" to the 
questions being asked. The disadvantage is that the 
responses you get might be "all over the lot" and not easy to 
summarize. You will likely have to sift through the range of 
responses to identify significant themes or patterns as they 
emerge from the responses. Other disadvantages include: ( 1) 
some respondents might not have the time, interest, or 
writing or oral language skills to respond meaningfully; and 
(2) narrative responses will be more difficult for the team to 
organize and analyze. 

As the name indicates, closed-ended questions narrow 
down the range of answers your respondents can give. Such 
questions allow you to specify ahead of time the answers 
which you are looking for or which you think are the most 
likely to emerge, making your analysis easier. For example, a 
close-ended question might ask: 

so 
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We would like to know if you have improved your 
communication skills on the iob as a result of 
participating in your class. Please circle all that apply: 

Since I started my class. I : 

(1) talk to my supervisor 

(2) read information on the bulletin boards 

(3) read safety materials 

(4) Other, please specify 

(If you pre-code your answers, always insert an "other" 
category where the respondent can supplement or 
personalize the pre-coded choices.) 

There are advantages to using close-ended questions, 
especially if you are surveying more than thirty people. You 
will be able to analyze your data more quickly if you have 
thought through possible answers ahead of time. 

Feel free to mix open- and close-ended questions in your 
survey, along with a few numbered scale items. Especially 
if this is the team's first evaluation, a mix of items will give 
team members the opportunity to experience the different 
types of answers which the various kinds of questions yield. 

Consider both "quantitative" and "qualitative" designs. 

As noted directly above and under Activity #6.b., you can 
design your information-gathering activities to produce 
"quantitative" information (presented in "numbers" or 
"statistics") or "qualitative" information (presented in 
"narrative" or "word" form). Close-ended questions tend to 
produce "quantitative" (readily "quantifiable" or 
"measurable") information, while open-ended questions tend 
to produce "qualitative" information. As noted above, each 
format for capturing information has advantages and 
disadvantages. 

Shown below are examples of two kinds of instruments 
designed to get feedback from a program participant about 
his or her uses of communication skills at work. The first is 
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a questionnaire which will produce information in a 
"quantitative" format; the second is an interview guide 
which will generate information which is more "qualitative" 
in nature. 

A "quantitative" questionnaire 

The following questionnaire is a pre-program ("baseline") 
version, to be used to collect information before the 
participants begin their class. The post-program version 
would include all the same questions, and any others which 
the team thinks are important to add after looking at the 
baseline responses. 



A "Quantitative" Questionnaire for Program Participants: 
Self-Assessment of Communication Skills at Work 

Please circle one answer for each question. 

♦ I say what's really on my mind in meetings 

very often often sometimes never 

♦ I make suggestions about how to improve the way we work 

very often often sometimes never 

♦ I ask my supervisors questions when I need to 

very often often sometimes never 

♦ I use the correct technical terms when I talk to my supervisors 

very often often sometimes never 

♦ I use the correct technical terms when I write memos 

very often often sometimes never 

A "qualitative" interview 

The following is an "interview guide n which an 
interviewer might use to elicit participants' feedback on the 
same general questions covered in the above more- 
structured questionnaire. However, this interview contains 
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open-ended questions which can allow participants to speak 
more spontaneously and freely about their experience. 



A "Qualitative" Interview for Program Participants : 
Self-Assessment of Communication Skills at Work 



1 . In what situations at work are you called upon to communicate with your 
supervisors and co-workers? 

2. In those situations, please describe what you typically talk or write about 
- and with whom. 

3. Tell me about any time you have had difficulty making yourself 
understood either in writing or when speaking. What was the problem? 

4. In meetings, what do you do when you have something you want to say? 
Are you generally able to say what's on your mind? Why or why not? 

5. At work, what do you do when you want to make a suggestion about how 
to improve how work is done? Do you generally make suggestions? Why 
or why not? 

6. Please describe what you do when you need to ask your supervisor a 
question. 

7. What do you do when you need to use correct technical terms when you 
talk with your supervisors? Is this a problem for you? Why or why not? 

8. When you write memos, how comfortable do you feel using technical 
terms? Please explain what terms you need to use and what you do. 



With the above considerations in mind, you ~ alone or with 
other team members ~ should now prepare a draft questionnaire. 
Different versions of this questionnaire would be completed by 
participants, union stewards, and supervisors. 

The questionnaire should ask the various information sources 
for feedback on the most important summative and formative 
questions you developed earlier in this Activity (See "Adapt your 
core questions for use with different sources and in various 
information-gathering activities." above.) While each version of 
the questionnaire should focus on essentially the same questions, 
the wording will differ slightly for each information source, to 
reflect their perspective relative to the program. Decide at this 
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point whether you want to produce information in a "quantitative" 
form, in a "qualitative" form, or some combination of the two. 

If you do this as a group, you might divide into two or more 
smaller groups. Each group can draft its own set of 
questionnaires, and then all the groups can share what they came 
up with and agree on a version they are most comfortable with. 

As you design these instruments, you might refer to sources 
cited in this Guide's bibliography for further guidance. 



ACTIVITY #7.b. 

ON YOUR OWN OR WITH THE TEAM : PREPARE 
ADDITIONAL DRAFT INSTRUMENTS. 

The discussion of the concept of "triangulation" in "Key Issues 
in Design of Information-Gathering Activities" (Activity #6.b.) 
noted that there is an advantage to asking similar questions of 
different groups of people in different information-gathering 
activities. The advantage is that, by asking similar questions of 
different sources, you will get a number of perspectives on that 
question. You can see where there is agreement — or 
disagreement ~ among different sources regarding any given 
question. 

In Activities #6.a. and #6.c, you mapped out a wide range of 
possible information sources and information-gathering activities. 
Review these options once again, and with the team decide which 
activities ~ in addition to those you just developed in Activity 
#7. a ~ would also be most appropriate for your evaluation. 
Remember that the purpose of using additional information- 
gathering activities is to collect any important summative and 
formative information not already captured in the activities you 
designed in Activity #7. a. 

Work with the team to develop drafts of plans ("instruments") 
for those activities. You might do this in several ways: 




• As facilitator, work on your own or with one or two other 
team members or colleagues from outside the team. Prepare 
drafts of instruments, get feedback from the team, and revise 
them. 

• In one or more meetings with all or part of the team, divide 
members into smaller groups. Each group can develop one or 
more draft instruments, and then the entire group can review 
those drafts and revise them. 

It is up to you and the group how many additional instruments 
to design and how members will work together to design them. 
The goal here is to (1) produce drafts of a reasonable number of 
information-gathering activities which they can field-test, (2) 
further develop team members' interest and expertise in the 
evaluation process. 

Again, if you or members feel you need additional guidance in 
the technicalities of designing information-gathering activities, 
you might refer to the bibliography and/or talk with others with 
experience designing such activities. 



ACTIVITY #7.c. 

WITH THE TEAM : AGREE ON A SET OF INFORMATION- 
GATHERING ACTIVITIES TO FIELD-TEST. 

After the team has developed drafts of several data-gathering 
instruments, it will have to agree on which options they will field- 
test. They should start out small, with one or two ways of 
collecting information - as a kind of test of their evaluation 
capacity. 

When making this decision for the field-test, the team should 
consider that one or more members will have to be responsible 
for collecting and analyzing information and reporting the results. 
Think through the tasks required and the resources available to 
the team. When you develop your evaluation action plan in Step 
#9, team members should be ready to commit themselves to 
whatever tasks they agree to be responsible for. 
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Step 8 : Field-test your instruments. 



ACTIVITY #8.a. 

WITH THE TEAM : PLAN YOUR FIELD-TESTS. 

With team members, go over the following points to be sure 
that all members understand the purposes and options for field- 
testing the drafts of information-gathering instruments they 
developed in Step 7: 

Purposes for conducting field-tests 

The team now has a set of draft instruments (that is, plans for 
possible information-gathering activities). But — as drafts — 
those instruments should be given a "field-test" before you use 
them fully. The team can do a trial run with fellow team 
members and/or with others outside the team, including a sample 
of those they eventually hope to gather information from. 

By so doing, the team will ( 1) clarify whether the wording or 
format of the instruments needs to be changed; (2) give those who 
will use the instrument a chance to better understand what they 
need to do to use them effectively; and (3) improve or establish 
the reliability and validity of the instrument — as discussed in 
Activity #6.b. 

What to look for in field-tests 

There are several ways of field-testing your instruments. In 
your field-tests, you should be looking for any problems or 
trouble spots in the instruments. As discussed in Activity #7.a, 
field testers should be sure that: 

~ Your instructions and overall presentation are clear; 

~ The language used is appropriate for those being asked to 
respond; 

-- Only one concept is dealt with in each question; 
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~ Possible responses are simple and clear. 

When field-testing instruments, members should ask 
themselves: 

- Do I feel "positive" about using these instruments? (Am I 

learning something useful? Am I showing respect to those 
I'm asking questions of?) 

- Am I getting the information I want from the appropriate 

source? 

-- Am I recording information in an accurate, non-judgmental 
way? 

~ What is this experience telling me about being an 
information-gatherer? 

~ What else do I need to do to use this instrument well? 

Possible ways of conducting field-tests 

Possible ways of field-testing your draft information-gathering 
instruments include: 

A "panel" reviews the instrument . This panel can be just two 
or three colleagues who know enough about the evaluation, the 
employees and supervisors, or what your program is trying to 
achieve to render a critical opinion about the instrument. Ask 
the panel members for feedback. 

Sample respondents do a "trial, run ." A few representatives of 
the various groups from whom the team will eventually get 
information, or others like them, participate in your activity 
and give your feedback on it. Because this group will match 
the group you actually evaluate as closely as possible, ask 
them: 

~ to explain any items they did not understand or had 
difficulty responding to; 

~ what they were thinking while they answered each item. 
Did the question trigger the information that the team 
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was looking for? If it's a long activity, just ask them 
what they were thinking on selected items with which 
you anticipate possible problems. 

Team members try the instrument out on each other . Make 
these trials as realistic as possible, asking those involved to 
answer each question with an open, critical mind. Switch roles 
and give all team members an opportunity to practice using the 
instruments, either in the role of the information-gatherer or of 
the person whom information is being collected from . The 
disadvantage to this method of field testing is that, if team 
members are already familiar with the instrument, they might 
not bring the critical perspective needed for a proper field test. 



ACTIVITY #8.b. 

WITH THE TEAM : CARRY OUT YOUR FIELD-TESTS AND 
LEARN FROM THEM. 

The team should now carry out the field-tests they decided on 
in Activity #8.a. As they get experience collecting information, 
members should collectively talk about the information they 
gathered and the process they went through using the various 
instruments. 

From these trials, you can decide: 

1. What if any revisions need to be made in the instruments. 

2. Whether any instruments should be discarded. 

3. Which team members should use which instruments. 

4. What additional preparations team members might need if 
they are to use the instruments effectively. 

If necessary, make changes in the instruments and give team 
members who will use them additional practice in using them. 
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Step 9 : Develop an action plan for an 
initial round of gathering, analyzing, 
and reporting information. 

You now have developed and tested several information- 
gathering activities. Now it's time to prepare an action plan to clarify 
how and when you will use those instruments in a coordinated effort 
over several months. 

You might limit your action plan to, say, a period of between 
three and six months. In that time, you can use your new 
instruments to collect information, analyze it, and then report your 
findings to selected audiences. See these initial evaluation activities 
as an opportunity to develop both information and expertise which 
you might in turn use as building blocks for a future, ongoing 
evaluation system. 



ACTIVITY #9.a. 

ON YOUR OWN : DRAFT AN ACTION PLAN FOR AN INITIAL 
ROUND OF EVALUATION ACTIVITIES. 

To prepare a draft of an action plan to show to the team, first 
re-exarnine each of the information-gathering activities the team 
decided on in Activity #7.c. For each activity, write down: 

• The name of the information-gathering activity. 

• The types of information it will collect. 

• The source(s) of that information. 

• Who will carry out the activity. 

• When the instrument will be used. 

Once you have outlined this information for each activity, step 
back and lay out a timeline for coordinating all the information- 
gathering. In that timeline, consider how often each activity will 
be conducted. (For example, will you conduct focus groups with 
learners only one time, or will you do so at the beginning, mid- 
point, and end of the program cycle?) 
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For each information-gathering activity, your plan should also 
specify how the resulting information will be used. Study the 
Steps outlined in Phase IV of this Guide. For each information- 
gathering activity, note who will be responsible for organizing the 
resulting information, and how it will be analyzed and reported. 
Shown below is a sample timeline for a six-month period: 



Sample timeline 
for an initial round of evaluation activities 





Jan. -Feb- 


March-April 


May-June 


Focus group 
with 

supervisors 


• Robert A. leads 
1st group 2nd 
week of Jan. 

• Maria H. 
analyzes data 3rd 
wk. of Jan. 

• Maria H. reports 
findings in 
monthly eval. 
team meetings. 


• Robert A. leads 
2nd group 3rd 
week of April. 

• Maria H. 
analyzes data last 
wk. of April. 

• Maria H. reports 
findings in 
monthly eval. 
team meetings. 


• Robert A. leads 
last group 1st 
week of June. 

• Maria H. 
analyzes data 2nd 
wk. of June. 

• Maria H. uses 
findings from 3 
groups in draft of 
final report. 


Focus group 
with learners 


• Jonathan A. 
leads 1st group 
2nd week of Jan. 

• Maria H. 
analyzes data 3rd 
wk. of Jan. 

• Maria H. reports 
findings in 
monthly eval. 
team meetings. 


• Jonathan A. 
leads 2nd group 
3rd week of April. 

• Maria H. 
analyzes data last 
wk. of April. 

• Maria H. reports 
findings in 
monthly eval. 
team meetings. 


• Jonathan A. 
leads last group 
1st week of June. 

• Maria H. 
analyzes data 2nd 
wk. of June. 

• Maria H. uses 
findings from 3 
groups in draft of 
final report. 


Feedback 
sessions with 
instructors 


• Walter D. leads 
feedback sessions 
in weekly staff 
meetings. He 
compiles minutes 
of those sessions. 

• Walter D. 
reports findings 
in monthly eval. 
team meetings. 


• Walter D. leads 
feedback sessions 
in weekly staff 
meetings. Me 
compiles minutes 
of those sessions. 

• Walter D. 
reports findings 
in monthly eval. 
team meetings. 


• Walter D. leads 
feedback sessions 
in weekly staff 
meetings. He 
compiles minutes 
of those sessions. 

• Walter D. 
reports findings 
in monthly eval. 
team meetings. 

• Maria H. uses 
those minutes in 
draft final report. 



You now have a draft action plan to show to the team. Because 
the implementation of the plan will require active participation by 
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team members, their input is vital to ensure that the plan is on 
target and that the team feels "ownership" for it. 



ACTIVITY #9.b. 

WITH THE TEAM : AGREE ON AN ACTION PLAN FOR THE 
COMING MONTHS. 

In a team meeting, explain that you now need to decide the 
details of using the instruments you've developed, who will use 
them to gather information, when, how the resulting information 
will be organized, and so forth. 

Explain that, based on your prior discussions with the team, 
you have drafted a strategy, or "action plan," for them to consider 
today. Explain how you prepared the plan and then show them 
your timeline. 

Go through each activity point-by-point, to clarify timing, 
responsibilities, and so forth. Explain that the team so far has not 
gotten into details of how to organize, analyze, and report the 
information to be collected, but that you will be meeting with 
them to facilitate those steps as you get to them. 

Be sure that no one feels coerced or confused by the plan. See 
that everyone feels comfortable with the roles they agree to play. 
Agree that, as the plan unfolds, the team will meet periodically to 
monitor progress and revise the plan as needed. 
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PHASE IV 

Gather, organize, analyze, 
and report the information. 

What happens in Phase IV 

In Step 10, the team will carry out its action plan and collect 
information from various sources. As this information is collected, 
they will organize it in ways which will make analysis of it easy. 

In Step 11, the team studies the information collected and 
interprets it, with a special emphasis on mapping out "next steps" ~ 
actions which now need to be taken regarding issues which have 
emerged. The information and the team's interpretations are 
summarized in a report. 

In Step 12, the team decides how it will convey key findings to 
its various intended audiences, carrying out any special preparations 
needed to ensure clear communication with those audiences. Team 
members then present their report - in written, oral, and/or taped 
formats. These presentations are carefully organized to ensure that 
the audiences consider the issues and actions which the team feels 
are important. 
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Step 10 : Gather and organize the 
information. 



In Phase III, the team prepared an action plan for an initial 
round of evaluation activities, along with specific information- 
gathering instruments to use. Now it's time to get on with 
implementing that plan. In this Step, the team will gather 
information and organize it so it can then be analyzed in Step 1 1. 

Here are some suggestions for doing so: 



/ 

ACTIVITY #10.a. 

WITH THE TEAM : GATHER INFORMATION. 

Those who have agreed to gather information should do so, as 
indicated in the action plan developed in Step 9. As they do so, 
they need to respect the groundrules the team has set for itself in 
terms of maintaining confidentiality, not being unnecessarily 
obtrusive, and so forth. 

Given the constantly-changing schedules of many workplaces, 
you will have to be flexible in terms of scheduling particular 
information-gathering activities. As necessary, revise your 
original timeline to respond to the realities of your information 
sources. Also be willing to revise the wording, the number of 
questions, and other aspects of your information-gathering 
activities as experience dictates. Team members might meet 
periodically once information-gathering gets underway, to iron 
out any additional issues which weren't dealt with in the field 
testing. 
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ACTIVITY #10. b. 



ON YOUR OWN OR WITH THE TEAM : ORGANIZE THE 
INFORMATION. 

As the team collects information, someone needs to keep it 
organized* If it isn't organized as it comes in, you might wind up 
with a pile of information which is difficult — if not impossible — 
to analyze. 

Organizing information is a "detail" task which some team 
members may not have time or interest to do. The facilitator or 
one or more other team members with time, interest, and 
creativity should organize the incoming information. Organizing 
information means gathering it in manageable, meaningful strands 
or themes. There are several approaches you can take: 

According to the major questions you want to answer : Those 
responding to your questionnaires, interviews, or focus groups 
will be giving you feedback on one or more of the core 
questions you wanted answered. If you asked for both 
"summative 11 feedback and "formative" feedback, you might 
mark all of the "summative" responses with a particular color 
or number and the "formative" information with another color 
or number. In this way, each set of core questions and their 
answers are easily identified. When you then analyze your 
data in the next Step, you can easily find all information 
pertaining to your major questions by looking for the 
corresponding color or number code. 

According to the source of the information : Prepare a number 
of folders marked with the names or titles of the various 
sources of the information coming in to you (for example: 
program participants, supervisors, union stewards, or 
instructors.) If you keep track of the sources of your 
information in this way, you will ~ when you get around to 
analyzing all of the data gathered be able to compare what 
supervisors say about a particular question with what other 
information sources say. 
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Once you have sorted out the information, sit down and read 
through it. You might read it once, to get a sense for the themes 
that emerge within and across your respondent groups. Then go 
back, read it more carefully a second time and take notes. 
Whoever takes responsibility for this part of the work - the 
facilitator or the facilitator and other team members -- will 
prepare a written summary of information for presentation to the 
whole team. To write the summary, do the following: 

• Identify themes . As you read through the information, 
certain themes are bound to emerge. Note them. Your first 
impressions of what your respondents are telling you are 
usually accurate. These themes provide a framework around 
which you can analyze the data in Step 11. 

For example, you might have asked participants if their 
communication skills on the job have improved. They answer 
"yes" and then further specify how the skills have improved, 
telling you: 

- I ask more questions. 

~ I share what I know with others. 

- I speak up in meetings. 
~ I write better memos. 

Consider these commonly-cited themes as "coding 
categories." 

• Quantify the number of times particular responses were 
given . Go back and calculate how many responses were given 
in each category. Simple "frequencies" of responses supply you 
with important information. They tell you how strong or weak 
a particular theme is, and signal whether you should pay close 
attention to the theme. 

• Maintain objectivity . Avoid inserting your interpretations at 
this point. (Or, if you do, be explicit that you are doing so by 
recording your interpretations ~ with your initials ~ in 
parentheses.) Try to keep your interpretations clearly 
separated from what your information sources said. 
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Step 11 : Analyze the information and 
prepare a report. 



Once the information has been laid out in the above ways, the 
team can take the next step of "analyzing" or "interpreting" it. The 
facilitator or a few team members might make some initial 
interpretations. They can then present the information ~ and their 
interpretations — to the rest of the team for their input. 



ACTIVITY #11. a. 

ON THEIR OWN : KEY MEMBERS INTERPRET THE 
INFORMATION. 

The facilitator or the facilitator and other team members who 
organized the data should take the lead in doing a prehminary 
analysis of the information. They should now carefully read 
through the summary of information which was prepared in 
Activity #10.b. For each section of information, each "reviewer" 
should note any comments, recommendations, or questions she or 
he might have in response. These notes should be titled 
something like "Facilitator's/Subcommittee's Preliminary 
Interpretation." 

This is the point where the team begins to "evaluate" 
("determine the value of," "judge," "examine," "make decisions 
about") the information they have spent all this time collecting. 
All persons involved need to recognize that they come to this Step 
with certain biases ("inclinations," "values," "perspectives," 
"outlooks"). Rather than hiding them or pretending they don't 
exist, everyone should make his or her perspectives explicit here. 
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ACTIVITY #11. b. 

WITH THE REST OF THE TEAM : INTERPREr THE 
INFORMATION. 

Circulate to the rest of the team the written summary of 
information prepared in Activity #10.b. (Don't show them the 
"Facilitator's/Subcommittee's Preliminary Interpretation" just 
yet.) At the end of each section of the summary, leave a space 
titled "Team Members' Interpretations." Ask the team members 
to now read through the summary carefully and jot down any 
interpretations (i.e., comments, ideas, or questions) they might 
have. (You might have to meet individually with team members 
who have difficulty with written documents, to explain to them 
orally what information was collected.) 

In a team meeting, go through each section of the information 
and ask team members to share the comments, recommendations, 
or questions they came up with. Record their responses on 
flipcharts titled "Team Members' Interpretations." 

Then share the "Facilitator's/Sub-Committee's Preliminary 
Interpretations" you came up in with in Activity #11. a. Compare 
them with the team members' interpretations on the flipcharts. 

Come to a consensus about how the team wants to interpret 
each section of the information. If no clear consensus can be 
reached about particular issues, note the range of interpretations 
represented in the team, including any significant differences of 
interpretation. See those differences not as conflicts but as 
honest, legitimate differences of opinion. 

As you record your team's interpretations, be sure to make it 
clear what, if any, actions the team would like to see happen in 
response to the information they've gathered. The team might 
want to make specific statements about whether the program 
should be continued and, if so, what actions are needed to 
improve various program components. 
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ACTIVITY #ll.c. 



ON YOUR OWN AND/OR WITH THE TEAM : PREPARE A 
DRAFT OF A, FINAL REPORT. 

The facilitator or subcommittee now has in hand a detailed 
summary of the information gathered, the team's interpretations 
of each section of the interpretation, and the facilitator's or 
subcommittee's own interpretations of each section. This 
summary is, by necessity, detailed, as you want to include all of 
the key findings and team members' thinking. 

However, many if not most of your audiences might not have 
the time to study every detail. They will pay attention only to a 
concise version of what you want to say to them. Now is the time 
to prepare an "executive summary" of your findings. 

To do so, first write up ~ as concisely as possible ~ the 
detailed summary of information and the interpretations which 
you and team members have noted for each section of the 
information. You might organize this summary into the following 
sections: 



Sample outline 
for a summary of 
information gathered and team members' interpretations 

I. Program goals : What stakeholders want the program to 
accomplish 

• What the information sources indicated 

• Team members' interpretations 

II. Anticipated and unanticipated outcomes to date : 

• What the information sources indicated 

• Team members' interpretations 

III. S trengths and needed improvements for each program 
component: 

• What the information sources indicated 

• Team members' interpretations 
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Once you have organized your findings and interpretations, go 
back through each section and pull out those which you feel your 
audiences should pay most attention to. Edit this into a 2-5 page 
summary and position it as the first few pages of your final 
written report. 

Your final report should thus begin with an executive 
summary, which is followed by a more-detailed summary of the 
information and team members' interpretations. You might also 
prepare a brief introduction which describes the purposes and 
methodology (the process you went through) for the report. For 
future reference, you might prepare appendices containing 
detailed summaries of information, sample information-gathering 
instruments, schedules of information-gathering activities, written 
statements from program participants, or other information of 
interest to your audiences. (Most audiences won't want to see all 
the detailed information you could put in appendices. However, it 
is nonetheless important to keep at least one detailed version "for 
the record" ~ especially in case your team or anyone else would 
like to borrow your tools or findings for other evaluation work in 
the future.) When all of this assembled, prepare a table of 
contents, a cover page, and a page titled "Acknowledgements" 
("thank you's" to the various people who helped with the 
evaluation). 

As you assemble the report, pay attention to graphics. A good 
word processing program can help you prepare an attractive, 
professional-looking report. Avoid a "dense," cluttered look by 
including plenty of "white space" around your text. 

Another important point: Use clear language. Keep sentences 
short. Avoid unnecessary long words and jargon. 

If you have the resources, ask a professional editor ~ perhaps 
someone in the communications department of your participating 
company, union, or education institution ~ to help with layout, 
graphics, and clear language. 
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ACTIVITY #11. d. 



WITH THE TEAM : PREPARE A FINAL VERSION OF THE 
REPORT. 

Before meeting with the team again, submit your draft report 
to members and ask for their feedback on both the content and 
format ("the look") of the report. Explain that now is the time to 
fine-tune the report before dissemination to your audiences. 

Remember that the final written report ~ especially the 
executive summary - is all that many of your audiences will 
know about your team's efforts. Ask team members to submit 
their editing suggestions to the facilitator or subcommittee. Then 
revise the draft to create a polished, clearly written, high-quality 
document which effectively conveys the team's message. 
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Step 12 : Report your findings. 



You have now taken pains to prepare a high-quality report, but 
you need to take an extra step of figuring out how best to present 
that report before actually making your presentations. 

Some audiences will have time only to review a brief written 
executive summary. Others might want to read a detailed report and 
meet with you to discuss it. Others might prefer you to convey your 
key findings in a face-to-face oral presentation rather than in 
written form. In some situations, more than one person — perhaps 
including program participants - can make the oral presentation. 
Take time now to figure out how best to convey the findings you've 
worked so hard to prepare. 



ACTIVITY #12.a. 

WITH THE TEAM : DECIDE HOW BEST TO CONVEY YOUR 
FINDINGS TO YOUR INTENDED AUDIENCES. 

Meet with the team to show them the final version of the 
report. Remind them that the purpose of all the work done by the 
team to this point was to produce information for use by various 
audiences. Display a flipchart titled "Possible Audiences," showing 
who the team identified as audiences back in Step 4. Ask the 
team to now revise that list as necessary, adding new audiences, 
deleting others. 

Ask them to identify which of those audiences they consider 
most important at this stage. Mark those audiences as "Primary" 
and the remaining audiences as "Secondary." 

Explain that there are a number of ways of presenting the 
information contained in the report prepared in Step 11. These 
"formats" include: 
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POSSIBLE FORMATS FOR PRESENTING 
YOUR FINDINGS 



Written versions 

• Full, detailed report 

• Executive summary of the full report 

• Charts containing key information 

• Transparencies containing key information (for use with 
overhead projector) 

Oral presentations 

• Oral, face-to-face presentations to small or larger 
audiences 

Taped p resentations 

• Videotaped presentations 

• Audio taped presentations 

• Slide-tape presentations 

Explain that, when conveying its findings to a particular 
audience, the team can take one of two strategies. In the first 
strategy, you can merely "send in the report" without any 
expectation that you will hear back from your audience or discuss 
it with them any further. (We might call this the "send only" 
strategy.) In the second strategy, you might use the report as a 
way of opening further dialogue with your audiences, presumably 
with the hope that they will consider taking some action you 
recommend in the report. (This second option might be termed 
the "send and discuss" strategy.) 

For some audiences, you might not have the time or interest to 
do much more than send in a report. In those cases, choose the 
"send only" strategy. For other audiences, you might want to 
actively build dialogue and further collaboration. In those cases, 
choose the "send and discuss" strategy. 

Explain that, to ensure that your findings are used most 
effectively by your various audiences, now is the time to do some 
strategizing for reporting your findings and any necessary follow- 
up action. Here is one way of doing so: 
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DEVELOPING A STRATEGY 
FOR REPORTING YOUR FINDINGS 
TO YOUR AUDIENCES 

1. For each of the "primary" audiences identified by the team, 
prepare a separate flipchart with the name of the audience at 
the top. Divide each into three columns titled "strategy," 
"format," and "timeline." 

2. For one of those primary audiences, decide: 

a. Strategy : Will you use the "send only" strategy or the 
"send and discuss" strategy? 

b. Format : Which of the above format(s) (written, oral, 
taped) will you use to convey your findings? 

c. Timeline : Who will take responsibility for putting the 
findings into the appropriate format, presenting the 
findings, and any further follow-up discussions or other 
actions? 

3. Repeat Step 2 for each of your other audiences, using 
separate flipcharts. 

4. When flipcharts have been prepared for each audience, tape 
•them next to each other on the wall. Review all the charts to 
clarify: 

a. How frequently is the same reporting format used? (This 
will, for example, let you know how many copies to make 
of the executive summary.) 

b. Are the various timelines "in sync" with each other? (For 
example, is one person trying to make too many 
presentations to too many different audiences in the same 
time period? Should all the audiences receive the report 
at the same time, to avoid having any audience feeling 
"slighted"?) 
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ACTIVITY #12.b. 



WITH THE TEAM : PREPARE YOUR PRESENTATIONS. 

The team now agrees who will prepare and make presentations 
for your various audiences. The facilitator might work with those 
individuals to do the necessary preparations. This is an 
opportunity for team members to demonstrate their ownership 
for the team process and for the education program. They can 
also "stretch" themselves to develop new communication skills. 

Each team member who has agreed to give a presentation 
should be supported to ensure that she or he has all documents, 
visual aides, and other resources (e.g., refreshments, meeting 
rooms) necessary. The facilitator and other experienced team 
members can help them develop a plan for the sessions in which 
they will present the findings. That plan should allow time for 
presenting key findings and for follow-up discussion. Here are 
some questions for team members to consider when preparing 
their presentations: 



PREPARING YOURSELF 
FOR 

MEETING YOUR AUDIENCES 

1. Where will the meeting take place? On the team's own "turf 
or somewhere else? 

2. What will be the agenda for the meeting? What are the 
objectives for the meeting itself? What do you hope will 
happen as a result of the meeting? 

3. Who will facilitate the meeting? 

4. Who will be spokesperson for your team? 

5. Should someone serve as "secretary" for the meeting, 
recording minutes either on paper or with a tape-recorder? 

6. Should someone help with taping up fhpcharts, running the 
overhead projector, or distributing handouts? 

7. Should one or more people play the role of "greeter"? 

8. Should someone be in charge of refreshments? 

9. Where should you sit? 

10. What clothes should you wear? 
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11. What supplies (e.g., handouts, charts, refreshments) do you 
need to have ready? 



If necessary, team members might do a "dress rehearsal" of 
their presentations. (If program participants are to be part of 
these presentations, they might practice their roles as a classroom 
exercise.) Members might also enlist the help of colleagues who 
are specialists in communications (e.g., public speaking, 
preparation of visual aides) to give guidance to those who will be 
making presentations. 



The exact steps you take to convey your findings to each of 
your audiences will vary, depending on the specifics of the strategies 
you developed in Activities #12. a and #12.b. Here are some general 
guidelines for you to adapt to each of your audiences: 



ACTIVITY #12.c. 

WITH THE TEAM : CIRCULATE THE WRITTEN REPORT TO 
YOUR AUDIENCES. 

Individual team members can now take responsibility for 
circulating a written version of their report to those audiences 
identified in Activity #12. a. For most audiences, the team will 
likely submit some form of written report. To those audiences, 
send either just the executive summary or a complete, detailed 
report. 

For those audiences for whom you decided in Activity #12. a. to 
take a "send only" approach, merely tell them ~ in a cover letter, 
in person, or by phone -- that "here is our report for your 
information, as promised." You may or may not want to add a 
general comment welcoming feedback or questions. 

For those audiences for whom you decided to use a "send and 
discuss" strategy, you might tell them "Here is our report. We 
would like to discuss it further with you." Ask them to read it 
carefully, jot down any comments or questions they have, and be 
prepared to meet you at a particular date to discuss the report. 
(You might even send a brief questionnaire containing specific 
questions you'd like them to consider when reading the report. 
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By filling out this questionnaire prior to meeting you, the 
audiences will be more likely to study the report carefully and 
pay attention to issues you think are important.) 



ACTIVITY #12.d. 

WITH THE TEAM : SET UP MEETINGS WITH YOUR 
AUDIENCES. 

For those audiences for whom you decided to use a "send and 
discuss" strategy, contact them and arrange to meet them in 
person. Explain that you'd like to get their feedback on the report 
which you sent them, and to discuss specific recommendations 
contained in the report. 

Agree on a suitable time and place for that meeting. Make sure 
that the "ingredients" you anticipated in Activity #12.b. are in 
place. (For example, pick a time and place convenient to the 
audience — so that they will actually show up. The location 
should have adequate seating, wall space for displaying flipcharts, 
an overhead projector, and other needed facilities. You might 
arrange to have refreshments served, as well.) 

In addition to making logistical arrangements, you and the 
other team members should go over the other plans you made in 
Activity #12.b. for making effective presentations. 



ACTIVITY #12.e. 

WITH THE TEAM : MEET WITH YOUR AUDIENCES. 

Carry out the plans you've now developed for discussing your 
findings with your audiences. In most cases, a meeting is held to: 

1. Help the audience to understand the key findings of the 
report. 

2. Build communication between your team and the audience. 

3. Get the audience to take one or more specific actions detailed 
in the report. 
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To ensure that such objectives are met, you should at the 
beginning of the meeting explain what you hope to accomplish. 
Emphasize your interest in deciding specific actions which might 
be taken in response to the report. Whoever takes the role of 
facilitating the meeting should keep the discussion focused on the 
meeting's objectives. 

To do so, you might organize the discussion around these 
questions: 

QUESTIONS AROUND WHICH A 
DISCUSSION OF YOUR REPORT MIGHT BE ORGANIZED 

1. What does the report say? If the audience has not 
already studied the report, you or another team member can 
summarize what is said in the report. 

2. What questions or comments does the audience have 
in response? For each section of the report, ask your 
audience to respond with any questions, comments, ideas, or 
recommendations they might have. 

3. What actions should be taken next? Ask the audience 
what actions should be taken next in response to the report. 
Time permitting, identify both long-term and short term 
actions. Clarify who will take those actions ~ and when. (If 
there is not enough time to get into details of planning specific 
actions in this meeting, agree to have another meeting to do so 
in the near future, while interest is still high.) 

Someone on the team should play the role of "recorder" or 
"secretary," recording what is said either on flipcharts or a 
notepad. These notes are important, as they serve as a record 
which can be used to plan follow-up actions in the next Step. 

The facilitator of the meeting should make sure that the 
meeting follows basic groundrules of respect (i.e., allowing 
everyone to have a say) and "continuous improvement" (i.e., team 
decision-making to strengthen the program). 

As necessary, repeat this process for any other audiences you 
would like to communicate your findings to. 
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P HASE V 
Decide what happens next. 



What happens in Phase V 

The team has now submitted its evaluation report to its 
audiences and presumably identified an agenda for follow-up 
actions. 



In Step 13, team members are pro-active in (1) clarifying who 
has agreed to do what, (2) implementing those actions which they've 
agreed to take responsibility for, and (3) monitoring how well the 
agreed-upon actions are being carried out. 

In Step 14, the team turns its evaluation expertise on itself. 
Members take a few minutes to look at the evaluation process they 
have just gone through, to decide whether they want to continue 
working as a team and, if so, how to continuously improve their work 
together. 
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Step 13 : Take follow-up action 



The team has now met with its audiences and likely decided to 
take some further actions. Now it's time to review the notes from 
the meetings held with their audiences, and be sure that the agreed- 
upon actions are actually carried out. 



ACTIVITY #13. a. 

WITH THE TEAM : PLAN FOLLOW-UP ACTIONS. 

On your own or with other team members, assemble the notes 
from the meetings you held with your various audiences. Prepare 
minutes for each of those meetings. You might organize the 
minutes within the structure of the three discussion questions 
suggested in Activity #12.d.: 

1. What did your report say? ' 

2. What questions or comments did the audience(s) have in 
response? 

3. What actions did the audience(s) decide should be taken 
next? 

The minutes should be as clear as possible about who agreed to 
take what actions. Present the minutes to the rest of the team, 
and agree on what actions team members now need to take 
themselves. Clarify who is responsible for what actions, when, 
and what help and resources they might need. Agree when the 
team will meet again to monitor progress. 

The team should also clarify how the team will follow up with 
the various audience representatives who agreed to take action. 
The team needs to decide who will monitor whether the audiences 
are doing what they agreed to do, and how the team might 
provide assistance to the audiences. 
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ACTIVITY #13.b. 



WITH THE TEAM : IMPLEMENT AND MONITOR FOLLOW-UP 
ACTIONS. 

Team members now need to take the follow-up actions they 
agreed upon in Activity #14.a. The team should meet periodically 
to monitor progress toward those actions. This monitoring of 
progress on follow-up actions might, in fact, be a key focal point 
for the next round of the team's evaluation activities. 
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Step 14 : Evaluate the evaluation. 

In the spirit of continuous improvement and self-reflection, the 
team should practice what it preaches and evaluate its own activities, 
Team members might be a bit tired of "evaluation" at this point, but 
they can nonetheless examine their own activities as follows: 

1. Ask team members to give feedback via a 
questionnaire or individual interview. In informal 
interviews and/or via questionnaires, ask members to respond to 
the following kinds of questions: 

• What was the team trying to achieve in the evaluation? 

• What in fact was achieved? 

• What did you personally get out of participating in the team? 

• What did you like about the team? 

• What didn't you like? 

• Should the team continue operating in the future? 

• How might the team be improved? 

If questionnaires are used, the team can decide whether 
members should sign them. 

2. Have another team member summarize the responses. 
A team member should prepare a brief summary of members' 
responses to the above questions to discuss at a future meeting. 

3. Discuss the responses. In a team meeting, discuss the 
summary of members' responses. Encourage team members to 
add further comments to build on what other members said. 

4. Decide on future evaluation work. Decide whether to 
continue operating as an evaluation team. If members agree to do 
so, map out a timeline in which members can go through a similar 
process to plan and carry out meaningful evaluation activities. 
Decide how those evaluation activities fit into the larger cycle of 
program activities. Also identify what actions members might 
take to enable themselves to participate actively on the team. 
(Members might, for example, do some additional reading about 
workplace education, get some training in conducting group 
discussions, or write regular attendance at team meetings into 
their normal work schedule.) 
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